Sep 16 14:07:49.318967 ip-10-0-132-170 systemd[1]: Starting Kubernetes Kubelet... Sep 16 14:07:49.657176 ip-10-0-132-170 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 14:07:49.657176 ip-10-0-132-170 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 14:07:49.657176 ip-10-0-132-170 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 14:07:49.657176 ip-10-0-132-170 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 14:07:49.657176 ip-10-0-132-170 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 14:07:49.659214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.659131 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 14:07:49.662615 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662592 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662619 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662623 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662627 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662630 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662633 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662636 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662639 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662642 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662645 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662647 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662651 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662655 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662658 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662661 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.662656 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662664 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662667 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662670 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662673 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662676 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662678 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662681 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662689 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662692 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662695 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662697 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662699 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662702 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662705 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662707 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662710 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662712 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662714 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662717 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662719 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.663018 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662722 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662725 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662727 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662730 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662732 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662735 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662737 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662739 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662742 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662744 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662746 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662751 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662756 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662759 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662762 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662764 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662767 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662770 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662773 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.663494 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662776 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662778 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662781 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662783 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662786 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662788 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662791 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662793 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662796 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662798 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662800 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662803 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662805 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662808 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662810 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662812 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662815 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662817 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662820 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662822 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.663979 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662825 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662828 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662831 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662834 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662837 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662840 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662843 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662846 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662848 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662851 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662853 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662856 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662858 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662861 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662863 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662867 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662870 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662873 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662876 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.664457 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662879 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662881 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662884 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662886 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662889 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.662892 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663549 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663556 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663559 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663561 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663564 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663566 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663569 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663571 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663574 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663577 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663580 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663582 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663585 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663588 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.664983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663591 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663594 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663596 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663599 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663618 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663621 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663623 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663626 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663628 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663631 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663634 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663636 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663639 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663641 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663644 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663647 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663649 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663652 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663655 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663657 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.665561 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663659 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663662 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663664 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663667 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663669 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663672 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663676 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663680 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663684 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663687 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663691 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663694 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663697 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663700 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663703 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663706 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663708 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663711 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663715 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.666069 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663718 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663721 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663724 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663727 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663730 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663732 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663735 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663738 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663741 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663743 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663746 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663748 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663751 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663754 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663756 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663758 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663762 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663765 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663769 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.666546 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663772 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663775 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663777 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663780 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663783 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663786 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663788 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663791 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663793 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663796 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663798 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663801 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663803 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663806 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663808 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663811 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663813 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663816 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663818 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663820 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.667029 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663823 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663825 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663828 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663830 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663832 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663835 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.663837 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664193 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664198 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664200 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664203 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664206 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664210 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664214 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664217 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664220 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664223 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664226 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664229 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.667535 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664232 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664235 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664237 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664240 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664242 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664245 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664247 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664250 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664252 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664255 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664257 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664259 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664262 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664264 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664267 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664270 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664272 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664274 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664277 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664280 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.668017 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664282 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664285 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664287 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664291 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664295 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664298 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664300 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664303 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664306 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664309 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664312 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664315 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664318 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664321 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664323 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664326 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664328 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664331 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664333 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.668512 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664336 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664338 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664341 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664343 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664345 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664348 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664350 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664353 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664356 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664358 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664360 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664363 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664365 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664368 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664370 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664372 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664375 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664380 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664382 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664384 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.668996 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664395 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664398 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664401 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664405 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664407 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664409 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664412 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664414 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664417 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664419 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664422 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664425 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664427 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664430 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664432 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664435 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664437 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664439 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664442 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664444 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.669484 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664446 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664449 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664451 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664454 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664456 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664459 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664462 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.664465 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664539 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664573 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664578 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664582 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664588 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664592 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664596 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664599 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664617 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664620 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664623 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664626 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664629 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664632 2579 flags.go:64] FLAG: --cgroup-root="" Sep 16 14:07:49.670005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664635 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664638 2579 flags.go:64] FLAG: --client-ca-file="" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664641 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664644 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664648 2579 flags.go:64] FLAG: --cluster-domain="" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664650 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664653 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664656 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664660 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664663 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664666 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664669 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664672 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664675 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664678 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664681 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664684 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664686 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664690 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664694 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664697 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664700 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664705 2579 flags.go:64] FLAG: --enable-server="true" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664708 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664712 2579 flags.go:64] FLAG: --event-burst="100" Sep 16 14:07:49.670535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664715 2579 flags.go:64] FLAG: --event-qps="50" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664718 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664721 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664724 2579 flags.go:64] FLAG: --eviction-hard="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664727 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664730 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664733 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664736 2579 flags.go:64] FLAG: --eviction-soft="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664738 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664741 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664744 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664747 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664750 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664752 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664755 2579 flags.go:64] FLAG: --feature-gates="" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664759 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664762 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664765 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664768 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664770 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664773 2579 flags.go:64] FLAG: --help="false" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664776 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664779 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664782 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 14:07:49.671148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664785 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664788 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664792 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664795 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664798 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664801 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664804 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664807 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664810 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664813 2579 flags.go:64] FLAG: --kube-reserved="" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664816 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664819 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664822 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664825 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664828 2579 flags.go:64] FLAG: --lock-file="" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664830 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664833 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664836 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664841 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664844 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664846 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664849 2579 flags.go:64] FLAG: --logging-format="text" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664852 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664855 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 14:07:49.671807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664858 2579 flags.go:64] FLAG: --manifest-url="" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664861 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664865 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664868 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664872 2579 flags.go:64] FLAG: --max-pods="110" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664874 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664877 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664880 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664883 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664886 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664890 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664893 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664900 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664904 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664907 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664910 2579 flags.go:64] FLAG: --pod-cidr="" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664913 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664918 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664921 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664924 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664928 2579 flags.go:64] FLAG: --port="10250" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664931 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664934 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ca8f1971d2f24bc" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664937 2579 flags.go:64] FLAG: --qos-reserved="" Sep 16 14:07:49.672854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664939 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664942 2579 flags.go:64] FLAG: --register-node="true" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664945 2579 flags.go:64] FLAG: --register-with-taints="" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664949 2579 flags.go:64] FLAG: --registry-burst="10" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664951 2579 flags.go:64] FLAG: --registry-qps="5" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664954 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664957 2579 flags.go:64] FLAG: --reserved-memory="" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664960 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664963 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664966 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664969 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664972 2579 flags.go:64] FLAG: --runonce="false" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664974 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664977 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664980 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664983 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664986 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664989 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664995 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.664998 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665001 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665004 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665008 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665011 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665014 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 14:07:49.673908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665017 2579 flags.go:64] FLAG: --system-cgroups="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665019 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665024 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665027 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665031 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665035 2579 flags.go:64] FLAG: --tls-min-version="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665037 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665040 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665043 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665046 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665049 2579 flags.go:64] FLAG: --v="2" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665053 2579 flags.go:64] FLAG: --version="false" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665057 2579 flags.go:64] FLAG: --vmodule="" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665060 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.665064 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665172 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665176 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665178 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665181 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665184 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665186 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665189 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665192 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.675074 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665194 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665197 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665201 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665203 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665206 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665209 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665212 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665215 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665217 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665220 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665223 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665225 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665227 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665230 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665233 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665235 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665238 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665241 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665243 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665246 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.676196 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665250 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665254 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665256 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665259 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665262 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665264 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665267 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665269 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665272 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665274 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665277 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665280 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665282 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665285 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665288 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665291 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665295 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665299 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665305 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.676897 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665308 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665311 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665313 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665316 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665319 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665321 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665324 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665327 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665330 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665333 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665335 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665338 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665340 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665344 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665348 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665351 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665354 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665356 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665359 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665362 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.677709 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665364 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665367 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665369 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665372 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665374 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665377 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665379 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665383 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665385 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665388 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665390 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665394 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665396 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665399 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665401 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665404 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665408 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665411 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665414 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.678428 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665417 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665420 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665423 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665425 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665428 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665430 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665433 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665435 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665438 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665440 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665443 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665445 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.665448 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.666096 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.674491 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 14:07:49.679086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.674513 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.674539 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.674551 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674662 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674671 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674676 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674681 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674686 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674690 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674695 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674700 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674705 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674709 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674714 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674718 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674722 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674726 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674733 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674739 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.679468 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674745 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674749 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674753 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674758 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674761 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674766 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674770 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674774 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674778 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674782 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674786 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674791 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674795 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674811 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674816 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674820 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674824 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674828 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674832 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674837 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.680038 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674841 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674845 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674849 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674853 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674860 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674866 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674870 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674875 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674880 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674885 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674890 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674894 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674899 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674904 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674908 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674912 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674916 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674920 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674925 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.680560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674929 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674933 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674937 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674940 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674945 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674949 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674954 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674966 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674971 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674975 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674981 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674987 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674991 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674995 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.674999 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675003 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675007 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675012 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675016 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.681068 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675020 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675024 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675028 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675033 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675037 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675041 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675046 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675050 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675054 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675058 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675062 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675066 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675070 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675073 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675077 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675081 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675085 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675089 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675092 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675097 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.681634 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675101 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675113 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675118 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675122 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675126 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.675134 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675386 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675395 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675400 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675404 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675409 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675413 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675418 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675425 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675431 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 14:07:49.682424 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675436 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675440 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675445 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675449 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675453 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675457 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675462 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675466 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675470 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675474 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675478 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675484 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675489 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675495 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675499 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675503 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675508 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675512 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675517 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 14:07:49.682937 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675530 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675535 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675540 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675544 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675548 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675552 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675556 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675560 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675564 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675568 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675572 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675576 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675580 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675585 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675589 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675593 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675597 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675620 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675625 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675628 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 14:07:49.683403 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675632 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675635 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675639 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675642 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675646 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675649 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675653 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675657 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675660 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675664 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675668 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675671 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675675 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675687 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675691 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675695 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675698 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675702 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675705 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675709 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 14:07:49.684093 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675713 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675717 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675720 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675724 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675728 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675733 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675737 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675743 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675749 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675753 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675757 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675761 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675765 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675769 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675773 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675777 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675781 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675785 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675790 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675794 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 14:07:49.684586 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675798 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675802 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675807 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675811 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675815 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675819 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675832 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675836 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675839 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675843 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:49.675849 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.675857 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.676026 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.678820 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 14:07:49.685082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.679646 2579 server.go:1019] "Starting client certificate rotation" Sep 16 14:07:49.685420 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.679753 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 14:07:49.685420 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.682315 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 14:07:49.685420 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.684714 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 14:07:49.689057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.689041 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 14:07:49.699829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.699813 2579 log.go:25] "Validated CRI v1 runtime API" Sep 16 14:07:49.702857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.702844 2579 log.go:25] "Validated CRI v1 image API" Sep 16 14:07:49.703687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.703666 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 14:07:49.707466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.707450 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 88e7dde7-39d6-41e8-8970-cf94530b7a74:/dev/nvme0n1p4 c7ed953a-8202-4fb4-bda8-35055f154e6e:/dev/nvme0n1p3] Sep 16 14:07:49.707532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.707465 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 14:07:49.712014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.711995 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 14:07:49.713426 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.713322 2579 manager.go:216] Machine: {Timestamp:2026-09-16 14:07:49.712181792 +0000 UTC m=+0.289542936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103541 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e7df8aae2e71f67eee78217c3173 SystemUUID:ec21e7df-8aae-2e71-f67e-ee78217c3173 BootID:700429e3-d4cb-4749-a714-9812868ce6a6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:bc:1a:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:bc:1a:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:20:0c:cf:3c:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 14:07:49.713426 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.713425 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 14:07:49.713529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.713495 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 14:07:49.715842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.715818 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 14:07:49.715962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.715845 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 14:07:49.716868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.716858 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 14:07:49.716901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.716870 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 14:07:49.716901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.716878 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 14:07:49.717510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.717501 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 14:07:49.718261 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.718252 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 14:07:49.722539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722528 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 16 14:07:49.722676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722667 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 14:07:49.722709 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722703 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 16 14:07:49.722741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722717 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 14:07:49.722741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722730 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 14:07:49.722741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722739 2579 kubelet.go:398] "Adding apiserver pod source" Sep 16 14:07:49.722829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.722746 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 14:07:49.723120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.723109 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 14:07:49.723181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.723127 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 14:07:49.726379 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.726361 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 14:07:49.728502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.728488 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 14:07:49.728556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.728519 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 14:07:49.730476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730465 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 14:07:49.730508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730482 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 14:07:49.730508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730488 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 14:07:49.730508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730492 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 14:07:49.730508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730503 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730510 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730523 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730528 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730535 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730540 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730548 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 14:07:49.730630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.730553 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 14:07:49.731254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.731245 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 14:07:49.731287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.731257 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 14:07:49.734792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.734779 2579 server.go:1268] "Started kubelet" Sep 16 14:07:49.734880 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.734858 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 14:07:49.734983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.734926 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 14:07:49.735088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.735001 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 14:07:49.735088 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.735041 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 14:07:49.735196 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.735088 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 14:07:49.735196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.735100 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 14:07:49.736218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.736202 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 14:07:49.736669 ip-10-0-132-170 systemd[1]: Started Kubernetes Kubelet. Sep 16 14:07:49.737632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.737451 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 16 14:07:49.739508 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.739491 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 14:07:49.739686 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.738834 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-170.ec2.internal.18d5d25068104801 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-170.ec2.internal,UID:ip-10-0-132-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-170.ec2.internal,},FirstTimestamp:2026-09-16 14:07:49.734754305 +0000 UTC m=+0.312115448,LastTimestamp:2026-09-16 14:07:49.734754305 +0000 UTC m=+0.312115448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-170.ec2.internal,}" Sep 16 14:07:49.741316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.741300 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 14:07:49.741758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.741743 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 14:07:49.741824 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.741813 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:49.742345 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742330 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 14:07:49.742438 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742356 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 14:07:49.742498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742332 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 14:07:49.742824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742727 2579 factory.go:55] Registering systemd factory Sep 16 14:07:49.742824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742776 2579 factory.go:223] Registration of the systemd container factory successfully Sep 16 14:07:49.742970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742951 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 16 14:07:49.742970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.742964 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 16 14:07:49.743105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743023 2579 factory.go:153] Registering CRI-O factory Sep 16 14:07:49.743105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743038 2579 factory.go:223] Registration of the crio container factory successfully Sep 16 14:07:49.743105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743084 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 14:07:49.743105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743103 2579 factory.go:103] Registering Raw factory Sep 16 14:07:49.743242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743119 2579 manager.go:1196] Started watching for new ooms in manager Sep 16 14:07:49.743425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.743410 2579 manager.go:349] Starting recovery of all containers Sep 16 14:07:49.748065 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.747905 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 14:07:49.748065 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.748011 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 14:07:49.756525 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.756383 2579 manager.go:354] Recovery completed Sep 16 14:07:49.760383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.760367 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.762181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762158 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.762257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762195 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.762257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762208 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.762812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762799 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.762878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762798 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 14:07:49.762878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762831 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 14:07:49.762878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.762852 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 16 14:07:49.763519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.763503 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.763596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.763528 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.763596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.763538 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.763875 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.763791 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-170.ec2.internal.18d5d25069b2c464 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-170.ec2.internal,UID:ip-10-0-132-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-170.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-170.ec2.internal,},FirstTimestamp:2026-09-16 14:07:49.762180196 +0000 UTC m=+0.339541341,LastTimestamp:2026-09-16 14:07:49.762180196 +0000 UTC m=+0.339541341,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-170.ec2.internal,}" Sep 16 14:07:49.765012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.764998 2579 policy_none.go:49] "None policy: Start" Sep 16 14:07:49.765056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.765020 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 14:07:49.765056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.765035 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 14:07:49.767132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.767121 2579 policy_none.go:47] "Start" Sep 16 14:07:49.767397 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.767384 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5dlrt" Sep 16 14:07:49.775366 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.775284 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-170.ec2.internal.18d5d25069b31c7e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-170.ec2.internal,UID:ip-10-0-132-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-170.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-170.ec2.internal,},FirstTimestamp:2026-09-16 14:07:49.76220275 +0000 UTC m=+0.339563892,LastTimestamp:2026-09-16 14:07:49.76220275 +0000 UTC m=+0.339563892,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-170.ec2.internal,}" Sep 16 14:07:49.776148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.776133 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5dlrt" Sep 16 14:07:49.805249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805224 2579 manager.go:337] "Starting Device Plugin manager" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.805282 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805301 2579 server.go:87] "Starting device plugin registration server" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805582 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805591 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805671 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805735 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.805742 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.806325 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.806379 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.808818 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.810145 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.810197 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.810242 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.810285 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 14:07:49.823497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.812115 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 14:07:49.906722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.906705 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.907725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.907661 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.907725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.907692 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.907725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.907701 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.907725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.907724 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.910809 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.910785 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal"] Sep 16 14:07:49.910893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.910856 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.911584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.911569 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.911670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.911600 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.911670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.911636 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.913990 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.913978 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.914105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.914146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914123 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.914716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914703 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.914825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914716 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.914825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914721 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.914825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914731 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.914924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914731 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.914924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.914873 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.916279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.916265 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.916344 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.916284 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-170.ec2.internal\": node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:49.917021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.917007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.917076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.917036 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:49.917746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.917733 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:49.917817 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.917755 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:49.917817 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.917764 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:49.934189 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.934169 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-170.ec2.internal\" not found" node="ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.938237 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.938224 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-170.ec2.internal\" not found" node="ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.944474 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.944456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.944527 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.944479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.944562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:49.944516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/579298bbec0b9e8e74fae2628d563376-config\") pod \"kube-apiserver-proxy-ip-10-0-132-170.ec2.internal\" (UID: \"579298bbec0b9e8e74fae2628d563376\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:49.953136 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:49.953119 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.044855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/579298bbec0b9e8e74fae2628d563376-config\") pod \"kube-apiserver-proxy-ip-10-0-132-170.ec2.internal\" (UID: \"579298bbec0b9e8e74fae2628d563376\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.044930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/579298bbec0b9e8e74fae2628d563376-config\") pod \"kube-apiserver-proxy-ip-10-0-132-170.ec2.internal\" (UID: \"579298bbec0b9e8e74fae2628d563376\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.044930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.044930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.044930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.044930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.044924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7554308d9b300ebecd820a92d9c922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal\" (UID: \"bf7554308d9b300ebecd820a92d9c922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.054050 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.054033 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.154699 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.154677 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.237135 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.237073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.240802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.240779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:50.255660 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.255642 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.356182 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.356134 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.456729 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.456690 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.557432 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.557369 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.657864 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.657837 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.680017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.679988 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 14:07:50.680154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.680137 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 14:07:50.739732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.739709 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 14:07:50.757948 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.757924 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.780093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.780065 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 14:02:50 +0000 UTC" deadline="2028-05-21 19:35:26.886337881 +0000 UTC" Sep 16 14:07:50.780093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.780089 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14717h27m36.10625229s" Sep 16 14:07:50.855018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.854943 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 14:07:50.856238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.856221 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 14:07:50.856346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.856250 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 14:07:50.856346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.856263 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 14:07:50.858345 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.858328 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.895759 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:50.895724 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7554308d9b300ebecd820a92d9c922.slice/crio-889f3ebdab99dc86658fcddef231c0f800c07a0bcb77b0ba37fd7d56463f548f WatchSource:0}: Error finding container 889f3ebdab99dc86658fcddef231c0f800c07a0bcb77b0ba37fd7d56463f548f: Status 404 returned error can't find the container with id 889f3ebdab99dc86658fcddef231c0f800c07a0bcb77b0ba37fd7d56463f548f Sep 16 14:07:50.901179 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.901162 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:07:50.959227 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:50.959200 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:50.987178 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:50.987156 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 14:07:51.059843 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.059826 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:51.160393 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.160328 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-170.ec2.internal\" not found" Sep 16 14:07:51.224751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.224730 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 14:07:51.242517 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.242497 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" Sep 16 14:07:51.256945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.256921 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 14:07:51.257746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.257733 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" Sep 16 14:07:51.274083 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.274064 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 14:07:51.524471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.524399 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 14:07:51.540214 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:51.540173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579298bbec0b9e8e74fae2628d563376.slice/crio-561b1cabc8c01c5bac2788d25452213bcb46be9147292080c43511245c428588 WatchSource:0}: Error finding container 561b1cabc8c01c5bac2788d25452213bcb46be9147292080c43511245c428588: Status 404 returned error can't find the container with id 561b1cabc8c01c5bac2788d25452213bcb46be9147292080c43511245c428588 Sep 16 14:07:51.724195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.724151 2579 apiserver.go:52] "Watching apiserver" Sep 16 14:07:51.732737 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.732710 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 14:07:51.734589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.734565 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zsbg2","openshift-dns/node-resolver-4rqpt","openshift-image-registry/node-ca-b97t4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal","openshift-multus/multus-additional-cni-plugins-xh5lp","openshift-multus/multus-l8mng","openshift-network-diagnostics/network-check-target-5cgs2","kube-system/global-pull-secret-syncer-rbr5h","kube-system/konnectivity-agent-k76cg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c","openshift-multus/network-metrics-daemon-7lx7g","openshift-network-operator/iptables-alerter-28htz","openshift-ovn-kubernetes/ovnkube-node-27422"] Sep 16 14:07:51.737510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.737492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.739631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.739599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.740949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.740929 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.740949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.740944 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vb8p8\"" Sep 16 14:07:51.741083 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.740947 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 14:07:51.741197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.741173 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.741197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.741183 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 14:07:51.741882 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.741862 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 14:07:51.742110 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.742094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.742737 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.742715 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 14:07:51.742837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.742761 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.742837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.742717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.742837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.742819 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vcqfc\"" Sep 16 14:07:51.744407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.744385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.744953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.744936 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 14:07:51.744953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.744944 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 14:07:51.745103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.744996 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 14:07:51.745103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.745011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wqqt2\"" Sep 16 14:07:51.745103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.744943 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.745427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.745334 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.746084 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.746066 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 14:07:51.747087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.747063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:51.747211 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.747165 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:07:51.747299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.747278 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.747946 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.747919 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.748111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.748085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 14:07:51.748208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.748157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gnrpc\"" Sep 16 14:07:51.750725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.750707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.750799 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.750772 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:07:51.753170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.753154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.754219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rks\" (UniqueName: \"kubernetes.io/projected/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-kube-api-access-l7rks\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.754219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-socket-dir-parent\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-conf-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-multus-certs\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-etc-kubernetes\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-ovn\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-config\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-script-lib\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-serviceca\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-cnibin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-bin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-slash\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-log-socket\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncxt\" (UniqueName: \"kubernetes.io/projected/e582fecb-01d3-4800-b116-f1e603c78edd-kube-api-access-4ncxt\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-host-slash\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-var-lib-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovn-node-metrics-cert\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxhd\" (UniqueName: \"kubernetes.io/projected/95fc6c93-c45e-4f95-9e0c-8f884706637b-kube-api-access-4lxhd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-hostroot\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-multus-daemon-config\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-iptables-alerter-script\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-kubelet\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-systemd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-bin\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-netd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-cni-binary-copy\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.754969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-kubelet\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.754980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-etc-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-node-log\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-system-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-k8s-cni-cncf-io\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-multus\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkmn\" (UniqueName: \"kubernetes.io/projected/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-kube-api-access-wxkmn\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-netns\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-env-overrides\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-host\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-systemd-units\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.755452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-netns\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.755933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.755911 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.756042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.756024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9hfbg\"" Sep 16 14:07:51.756101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.756064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.757308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.757289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.758457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.758434 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 14:07:51.759788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.759770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.759883 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.759840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:07:51.760016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.759997 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.760074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.760036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 14:07:51.760074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.759999 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.760142 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.760039 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zrgz\"" Sep 16 14:07:51.762143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.762122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.764663 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.764646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.764941 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.764917 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c9z8x\"" Sep 16 14:07:51.765030 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.765013 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 14:07:51.765090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.765076 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 14:07:51.767048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.767030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.767762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.767746 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 14:07:51.768158 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.768013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqqx6\"" Sep 16 14:07:51.768437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.768412 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 14:07:51.769778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.769759 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 14:07:51.770331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.770310 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p4grs\"" Sep 16 14:07:51.770428 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.770350 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 14:07:51.814534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.814493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" event={"ID":"579298bbec0b9e8e74fae2628d563376","Type":"ContainerStarted","Data":"561b1cabc8c01c5bac2788d25452213bcb46be9147292080c43511245c428588"} Sep 16 14:07:51.815515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.815497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" event={"ID":"bf7554308d9b300ebecd820a92d9c922","Type":"ContainerStarted","Data":"889f3ebdab99dc86658fcddef231c0f800c07a0bcb77b0ba37fd7d56463f548f"} Sep 16 14:07:51.825964 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.825944 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f4tq2" Sep 16 14:07:51.844025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.844008 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 14:07:51.848163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.848149 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f4tq2" Sep 16 14:07:51.855417 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-script-lib\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.855471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-serviceca\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.855471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-cnibin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.855550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhddv\" (UniqueName: \"kubernetes.io/projected/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-kube-api-access-lhddv\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.855550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-registration-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.855631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.855631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-cnibin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.855631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-slash\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-slash\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-log-socket\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-log-socket\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncxt\" (UniqueName: \"kubernetes.io/projected/e582fecb-01d3-4800-b116-f1e603c78edd-kube-api-access-4ncxt\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-socket-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-sys-fs\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.855822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqr64\" (UniqueName: \"kubernetes.io/projected/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-kube-api-access-sqr64\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-host-slash\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-var-lib-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-serviceca\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-host-slash\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-var-lib-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.855974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxhd\" (UniqueName: \"kubernetes.io/projected/95fc6c93-c45e-4f95-9e0c-8f884706637b-kube-api-access-4lxhd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-hostroot\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-multus-daemon-config\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-hostroot\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-script-lib\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.856579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-iptables-alerter-script\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.856579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-bin\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856652 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9hg\" (UniqueName: \"kubernetes.io/projected/87d538d9-4506-4269-8ec6-5d4d1c8e402a-kube-api-access-wc9hg\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.856652 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.856652 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysconfig\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.856652 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2x5p\" (UniqueName: \"kubernetes.io/projected/6ae28b05-e749-47b2-afb3-51fee098588b-kube-api-access-v2x5p\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-bin\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-system-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-k8s-cni-cncf-io\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-kubernetes\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-k8s-cni-cncf-io\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.856824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkmn\" (UniqueName: \"kubernetes.io/projected/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-kube-api-access-wxkmn\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-netns\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-host\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-system-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-host\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-netns\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.856973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-systemd-units\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d538d9-4506-4269-8ec6-5d4d1c8e402a-tmp-dir\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-systemd\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.857064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-systemd-units\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-system-cni-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rks\" (UniqueName: \"kubernetes.io/projected/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-kube-api-access-l7rks\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-run-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-iptables-alerter-script\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-socket-dir-parent\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-multus-certs\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-device-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.857431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-socket-dir-parent\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-ovn\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-config\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-bin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6fc\" (UniqueName: \"kubernetes.io/projected/d4e2dd42-2c87-474b-897a-f49c2efc39e1-kube-api-access-ll6fc\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857744 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-multus-daemon-config\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-bin\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.857869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-dbus\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-ovn\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.857998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-run\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-konnectivity-ca\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cnibin\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-cni-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovn-node-metrics-cert\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-kubelet\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-systemd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-netd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-kubelet\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-cni-binary-copy\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovnkube-config\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-systemd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-host-cni-netd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-multus-certs\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-kubelet\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-modprobe-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-etc-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-kubelet\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.858594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-node-log\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-multus\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-run-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-lib-modules\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-etc-openvswitch\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-var-lib-kubelet\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95fc6c93-c45e-4f95-9e0c-8f884706637b-node-log\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-tuned\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-env-overrides\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-var-lib-cni-multus\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87d538d9-4506-4269-8ec6-5d4d1c8e402a-hosts-file\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-agent-certs\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858874 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-netns\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.858904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e582fecb-01d3-4800-b116-f1e603c78edd-cni-binary-copy\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-host-run-netns\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.859386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-sys\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-host\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-kubelet-config\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-conf-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-etc-kubernetes\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-multus-conf-dir\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-conf\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e582fecb-01d3-4800-b116-f1e603c78edd-etc-kubernetes\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-tmp\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95fc6c93-c45e-4f95-9e0c-8f884706637b-env-overrides\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.860127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.859339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.863303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.863284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95fc6c93-c45e-4f95-9e0c-8f884706637b-ovn-node-metrics-cert\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.866532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.866512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkmn\" (UniqueName: \"kubernetes.io/projected/ae4ab7a4-b522-4b58-8b04-3379f14e2b3e-kube-api-access-wxkmn\") pod \"iptables-alerter-28htz\" (UID: \"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e\") " pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:51.866871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.866762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncxt\" (UniqueName: \"kubernetes.io/projected/e582fecb-01d3-4800-b116-f1e603c78edd-kube-api-access-4ncxt\") pod \"multus-l8mng\" (UID: \"e582fecb-01d3-4800-b116-f1e603c78edd\") " pod="openshift-multus/multus-l8mng" Sep 16 14:07:51.866939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.866928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxhd\" (UniqueName: \"kubernetes.io/projected/95fc6c93-c45e-4f95-9e0c-8f884706637b-kube-api-access-4lxhd\") pod \"ovnkube-node-27422\" (UID: \"95fc6c93-c45e-4f95-9e0c-8f884706637b\") " pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:51.868457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.868437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rks\" (UniqueName: \"kubernetes.io/projected/54f5582a-0f28-4fbd-a29b-cc8b94f968b1-kube-api-access-l7rks\") pod \"node-ca-b97t4\" (UID: \"54f5582a-0f28-4fbd-a29b-cc8b94f968b1\") " pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:51.960580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-modprobe-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-lib-modules\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-var-lib-kubelet\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-tuned\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87d538d9-4506-4269-8ec6-5d4d1c8e402a-hosts-file\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-agent-certs\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-var-lib-kubelet\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-lib-modules\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-modprobe-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87d538d9-4506-4269-8ec6-5d4d1c8e402a-hosts-file\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.960804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-sys\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-host\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-d\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-kubelet-config\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-host\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-conf\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960912 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-sys\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-tmp\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-kubelet-config\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.960990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhddv\" (UniqueName: \"kubernetes.io/projected/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-kube-api-access-lhddv\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-registration-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysctl-conf\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-socket-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-sys-fs\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.961545 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.961134 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-sys-fs\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-registration-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-socket-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.961257 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:07:52.461205749 +0000 UTC m=+3.038566898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqr64\" (UniqueName: \"kubernetes.io/projected/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-kube-api-access-sqr64\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9hg\" (UniqueName: \"kubernetes.io/projected/87d538d9-4506-4269-8ec6-5d4d1c8e402a-kube-api-access-wc9hg\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysconfig\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2x5p\" (UniqueName: \"kubernetes.io/projected/6ae28b05-e749-47b2-afb3-51fee098588b-kube-api-access-v2x5p\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-kubernetes\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-sysconfig\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.962476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d538d9-4506-4269-8ec6-5d4d1c8e402a-tmp-dir\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-kubernetes\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d538d9-4506-4269-8ec6-5d4d1c8e402a-tmp-dir\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.961907 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-systemd\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.961975 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:52.461957154 +0000 UTC m=+3.039318302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.961992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-systemd\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-system-cni-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-device-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6fc\" (UniqueName: \"kubernetes.io/projected/d4e2dd42-2c87-474b-897a-f49c2efc39e1-kube-api-access-ll6fc\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-dbus\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-system-cni-dir\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-device-dir\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.963313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-run\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-konnectivity-ca\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ae28b05-e749-47b2-afb3-51fee098588b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cnibin\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-cnibin\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e2dd42-2c87-474b-897a-f49c2efc39e1-run\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962453 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb787601-e0a7-43ca-a6c9-6f66b529995b-dbus\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.962803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-konnectivity-ca\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.963546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-etc-tuned\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.963917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.963562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e2dd42-2c87-474b-897a-f49c2efc39e1-tmp\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:51.964327 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.964305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34-agent-certs\") pod \"konnectivity-agent-k76cg\" (UID: \"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34\") " pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:51.971942 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.971914 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:07:51.971942 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.971940 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:07:51.972126 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.971954 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:51.972126 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:51.972020 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:52.472002146 +0000 UTC m=+3.049363298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:51.972257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.972233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2x5p\" (UniqueName: \"kubernetes.io/projected/6ae28b05-e749-47b2-afb3-51fee098588b-kube-api-access-v2x5p\") pod \"aws-ebs-csi-driver-node-c4k5c\" (UID: \"6ae28b05-e749-47b2-afb3-51fee098588b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:51.972422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.972402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9hg\" (UniqueName: \"kubernetes.io/projected/87d538d9-4506-4269-8ec6-5d4d1c8e402a-kube-api-access-wc9hg\") pod \"node-resolver-4rqpt\" (UID: \"87d538d9-4506-4269-8ec6-5d4d1c8e402a\") " pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:51.972491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.972462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhddv\" (UniqueName: \"kubernetes.io/projected/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-kube-api-access-lhddv\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:51.973398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.973376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqr64\" (UniqueName: \"kubernetes.io/projected/0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5-kube-api-access-sqr64\") pod \"multus-additional-cni-plugins-xh5lp\" (UID: \"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5\") " pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:51.974194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:51.974165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6fc\" (UniqueName: \"kubernetes.io/projected/d4e2dd42-2c87-474b-897a-f49c2efc39e1-kube-api-access-ll6fc\") pod \"tuned-zsbg2\" (UID: \"d4e2dd42-2c87-474b-897a-f49c2efc39e1\") " pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:52.051252 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.051189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8mng" Sep 16 14:07:52.058837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.058815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28htz" Sep 16 14:07:52.061889 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.061869 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode582fecb_01d3_4800_b116_f1e603c78edd.slice/crio-cf5dec546426f750da6642a7dc2cc65e5c247c3874735a41156e6f5ea9b99085 WatchSource:0}: Error finding container cf5dec546426f750da6642a7dc2cc65e5c247c3874735a41156e6f5ea9b99085: Status 404 returned error can't find the container with id cf5dec546426f750da6642a7dc2cc65e5c247c3874735a41156e6f5ea9b99085 Sep 16 14:07:52.065965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.065950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:07:52.067627 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.067606 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4ab7a4_b522_4b58_8b04_3379f14e2b3e.slice/crio-27e73def9eaf7d22d459c49ecfed1c72fdf62792bc7e2f5f8dd93f4473e62d2b WatchSource:0}: Error finding container 27e73def9eaf7d22d459c49ecfed1c72fdf62792bc7e2f5f8dd93f4473e62d2b: Status 404 returned error can't find the container with id 27e73def9eaf7d22d459c49ecfed1c72fdf62792bc7e2f5f8dd93f4473e62d2b Sep 16 14:07:52.070769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.070746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b97t4" Sep 16 14:07:52.075610 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.075589 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fc6c93_c45e_4f95_9e0c_8f884706637b.slice/crio-46eb0e456642974c7fe365160686e527fd490e616919139cf2d4cbf103c6d888 WatchSource:0}: Error finding container 46eb0e456642974c7fe365160686e527fd490e616919139cf2d4cbf103c6d888: Status 404 returned error can't find the container with id 46eb0e456642974c7fe365160686e527fd490e616919139cf2d4cbf103c6d888 Sep 16 14:07:52.077419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.077403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" Sep 16 14:07:52.082037 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.082016 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f5582a_0f28_4fbd_a29b_cc8b94f968b1.slice/crio-63388c3c069eef599f34027570dd3be3c087e85e09bee0f77691bba7e9ee95f0 WatchSource:0}: Error finding container 63388c3c069eef599f34027570dd3be3c087e85e09bee0f77691bba7e9ee95f0: Status 404 returned error can't find the container with id 63388c3c069eef599f34027570dd3be3c087e85e09bee0f77691bba7e9ee95f0 Sep 16 14:07:52.082728 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.082672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" Sep 16 14:07:52.086296 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.086277 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e2dd42_2c87_474b_897a_f49c2efc39e1.slice/crio-eec4bf1556eb75b3ba6dfd47b79c8bf02990ec589750b8eaaf4a9deae098ff50 WatchSource:0}: Error finding container eec4bf1556eb75b3ba6dfd47b79c8bf02990ec589750b8eaaf4a9deae098ff50: Status 404 returned error can't find the container with id eec4bf1556eb75b3ba6dfd47b79c8bf02990ec589750b8eaaf4a9deae098ff50 Sep 16 14:07:52.088303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.088287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:07:52.091518 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.091499 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae28b05_e749_47b2_afb3_51fee098588b.slice/crio-e520ce25840369ff722a25caa253700a0be45910b87faf7c8326f9aa18a1eefc WatchSource:0}: Error finding container e520ce25840369ff722a25caa253700a0be45910b87faf7c8326f9aa18a1eefc: Status 404 returned error can't find the container with id e520ce25840369ff722a25caa253700a0be45910b87faf7c8326f9aa18a1eefc Sep 16 14:07:52.095003 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.094981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" Sep 16 14:07:52.096798 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.096774 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode259e0c0_e3f5_4d9a_aa75_4b7716a0ed34.slice/crio-0bfd4be79110463e16255a4a319bc8be5da31a2785cb935d1385d060cf8787a3 WatchSource:0}: Error finding container 0bfd4be79110463e16255a4a319bc8be5da31a2785cb935d1385d060cf8787a3: Status 404 returned error can't find the container with id 0bfd4be79110463e16255a4a319bc8be5da31a2785cb935d1385d060cf8787a3 Sep 16 14:07:52.098996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.098800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4rqpt" Sep 16 14:07:52.105202 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.105183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca46f1c_db97_4cf0_a4f1_3c4e1bafcaf5.slice/crio-2ca0cf2a243c89fed33ddd01086b3f979ac13879164ecbba1e4fc8b20c7ff64f WatchSource:0}: Error finding container 2ca0cf2a243c89fed33ddd01086b3f979ac13879164ecbba1e4fc8b20c7ff64f: Status 404 returned error can't find the container with id 2ca0cf2a243c89fed33ddd01086b3f979ac13879164ecbba1e4fc8b20c7ff64f Sep 16 14:07:52.108616 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:07:52.108564 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d538d9_4506_4269_8ec6_5d4d1c8e402a.slice/crio-bc5b5bc11ae51dcbe105dcc987432bfc5858d5c35b94fb8f8d70a1162d7e441d WatchSource:0}: Error finding container bc5b5bc11ae51dcbe105dcc987432bfc5858d5c35b94fb8f8d70a1162d7e441d: Status 404 returned error can't find the container with id bc5b5bc11ae51dcbe105dcc987432bfc5858d5c35b94fb8f8d70a1162d7e441d Sep 16 14:07:52.466642 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.466536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:52.466642 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.466600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:52.466829 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.466713 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:52.466829 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.466717 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:52.466829 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.466769 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:07:53.466755213 +0000 UTC m=+4.044116344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:52.466829 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.466786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:53.466778615 +0000 UTC m=+4.044139746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:52.568203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.568158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:52.568387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.568311 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:07:52.568387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.568328 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:07:52.568387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.568338 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:52.568486 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:52.568396 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:53.568379562 +0000 UTC m=+4.145740701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:52.818177 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.818083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerStarted","Data":"2ca0cf2a243c89fed33ddd01086b3f979ac13879164ecbba1e4fc8b20c7ff64f"} Sep 16 14:07:52.819233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.819211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k76cg" event={"ID":"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34","Type":"ContainerStarted","Data":"0bfd4be79110463e16255a4a319bc8be5da31a2785cb935d1385d060cf8787a3"} Sep 16 14:07:52.820359 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.820334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" event={"ID":"d4e2dd42-2c87-474b-897a-f49c2efc39e1","Type":"ContainerStarted","Data":"eec4bf1556eb75b3ba6dfd47b79c8bf02990ec589750b8eaaf4a9deae098ff50"} Sep 16 14:07:52.821495 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.821476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8mng" event={"ID":"e582fecb-01d3-4800-b116-f1e603c78edd","Type":"ContainerStarted","Data":"cf5dec546426f750da6642a7dc2cc65e5c247c3874735a41156e6f5ea9b99085"} Sep 16 14:07:52.822674 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.822641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" event={"ID":"6ae28b05-e749-47b2-afb3-51fee098588b","Type":"ContainerStarted","Data":"e520ce25840369ff722a25caa253700a0be45910b87faf7c8326f9aa18a1eefc"} Sep 16 14:07:52.823778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.823748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b97t4" event={"ID":"54f5582a-0f28-4fbd-a29b-cc8b94f968b1","Type":"ContainerStarted","Data":"63388c3c069eef599f34027570dd3be3c087e85e09bee0f77691bba7e9ee95f0"} Sep 16 14:07:52.824915 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.824894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"46eb0e456642974c7fe365160686e527fd490e616919139cf2d4cbf103c6d888"} Sep 16 14:07:52.825986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.825968 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28htz" event={"ID":"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e","Type":"ContainerStarted","Data":"27e73def9eaf7d22d459c49ecfed1c72fdf62792bc7e2f5f8dd93f4473e62d2b"} Sep 16 14:07:52.827110 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.827091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4rqpt" event={"ID":"87d538d9-4506-4269-8ec6-5d4d1c8e402a","Type":"ContainerStarted","Data":"bc5b5bc11ae51dcbe105dcc987432bfc5858d5c35b94fb8f8d70a1162d7e441d"} Sep 16 14:07:52.851161 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.851116 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:02:52 +0000 UTC" deadline="2028-05-01 03:54:53.951552453 +0000 UTC" Sep 16 14:07:52.851219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:52.851173 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14221h47m1.100399155s" Sep 16 14:07:53.477130 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.477090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:53.477331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.477160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:53.477331 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.477299 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:53.477440 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.477369 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:55.477348417 +0000 UTC m=+6.054709571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:53.477833 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.477810 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:53.477941 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.477867 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:07:55.477849445 +0000 UTC m=+6.055210578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:53.578986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.578281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:53.578986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.578489 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:07:53.578986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.578515 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:07:53.578986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.578528 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:53.578986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.578590 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:55.578570673 +0000 UTC m=+6.155931823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:53.812501 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.812467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:53.812723 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.812598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:07:53.812916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.812877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:53.813048 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.812990 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:07:53.813048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.813038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:53.813161 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:53.813142 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:07:53.852379 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.852303 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:02:52 +0000 UTC" deadline="2028-03-07 14:54:08.249675307 +0000 UTC" Sep 16 14:07:53.852379 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:53.852346 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12912h46m14.397334746s" Sep 16 14:07:55.493468 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.493436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:55.494009 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.493480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:55.494009 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.493575 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:55.494009 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.493623 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:59.493608035 +0000 UTC m=+10.070969168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:55.494009 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.493781 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:55.494009 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.493847 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:07:59.493826163 +0000 UTC m=+10.071187314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:55.594436 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.594403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:55.594597 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.594547 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:07:55.594597 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.594563 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:07:55.594597 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.594572 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:55.594770 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.594628 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:07:59.594609314 +0000 UTC m=+10.171970468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.813774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.813904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.814290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:55.814327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.814402 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:07:55.814565 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:55.814481 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:07:57.810984 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:57.810951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:57.811425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:57.811001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:57.811425 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:57.811120 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:07:57.811425 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:57.811198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:07:57.811425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:57.811240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:57.811425 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:57.811288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.527901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.527971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.528119 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.528183 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:07.528161705 +0000 UTC m=+18.105522856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.528602 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:59.528712 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.528657 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:08:07.528640642 +0000 UTC m=+18.106001789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:07:59.628930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.628891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:59.629124 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.629061 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:07:59.629124 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.629086 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:07:59.629124 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.629099 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:59.629272 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.629160 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:07.629141523 +0000 UTC m=+18.206502656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:07:59.812623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.812518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:07:59.812783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.812632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:07:59.812783 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.812646 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:07:59.812783 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.812749 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:07:59.812920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:07:59.812794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:07:59.812920 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:07:59.812856 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:01.812453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:01.811717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:01.812453 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:01.811855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:01.812453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:01.812297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:01.812453 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:01.812404 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:01.813374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:01.813233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:01.813374 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:01.813340 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:03.810742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:03.810710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:03.811446 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:03.810845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:03.811446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:03.810715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:03.811446 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:03.810948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:03.811446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:03.810709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:03.811446 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:03.811025 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:05.810902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:05.810866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:05.811368 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:05.810991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:05.811460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:05.811442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:05.811548 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:05.811531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:05.811622 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:05.811581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:05.811688 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:05.811661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:07.591843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.591810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:07.592333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.591900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:07.592333 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.591987 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:08:07.592333 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.592065 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:23.592042674 +0000 UTC m=+34.169403804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:08:07.592333 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.591993 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:08:07.592333 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.592122 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:08:23.592108648 +0000 UTC m=+34.169469778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:08:07.693244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.693165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:07.693391 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.693342 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:08:07.693391 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.693368 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:08:07.693391 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.693378 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:08:07.693518 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.693435 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:23.69341528 +0000 UTC m=+34.270776410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:08:07.810651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.810618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:07.810651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.810655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:07.810898 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:07.810621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:07.810898 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.810763 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:07.810898 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.810834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:07.811041 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:07.810944 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:09.813360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:09.813315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:09.813812 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:09.813526 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:09.813982 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:09.813913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:09.815043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:09.814249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:09.815043 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:09.814253 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:09.815043 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:09.814459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:11.810961 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:11.810922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:11.810961 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:11.810937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:11.811439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:11.810930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:11.811439 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:11.811040 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:11.811439 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:11.811144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:11.811439 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:11.811269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:12.866103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.865723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k76cg" event={"ID":"e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34","Type":"ContainerStarted","Data":"87c6b4a09bdd2690fb725df8bc611ad918bbb6cf8115d22a35993da9a708640d"} Sep 16 14:08:12.867437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.867408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" event={"ID":"d4e2dd42-2c87-474b-897a-f49c2efc39e1","Type":"ContainerStarted","Data":"046ecc7d14b81790092555b4d477d609d0e6045a9aaf0d5307b85ade2d518335"} Sep 16 14:08:12.869130 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.869101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8mng" event={"ID":"e582fecb-01d3-4800-b116-f1e603c78edd","Type":"ContainerStarted","Data":"5ba7fe4896d9c7d107bedb11fb0d09f0d569add401d02332dc2b5855d4e2f880"} Sep 16 14:08:12.871470 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.871437 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf7554308d9b300ebecd820a92d9c922" containerID="49866142373bcd4e4d13c4f883d63a6b58ae832316e0fd6a898b90740f4881dc" exitCode=0 Sep 16 14:08:12.871586 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.871516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" event={"ID":"bf7554308d9b300ebecd820a92d9c922","Type":"ContainerDied","Data":"49866142373bcd4e4d13c4f883d63a6b58ae832316e0fd6a898b90740f4881dc"} Sep 16 14:08:12.873235 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.873215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" event={"ID":"6ae28b05-e749-47b2-afb3-51fee098588b","Type":"ContainerStarted","Data":"346beeac701d500583cc385bcaf9b24d9fc2d3d371a67b621a4f0c4acfabbad6"} Sep 16 14:08:12.878391 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.878203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b97t4" event={"ID":"54f5582a-0f28-4fbd-a29b-cc8b94f968b1","Type":"ContainerStarted","Data":"c03a1644e845502f9d783d92d0d4e8ff7c8fa528c2e899a01051cc0bcdf1bd8a"} Sep 16 14:08:12.881853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.881819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k76cg" podStartSLOduration=10.141320405 podStartE2EDuration="21.881807013s" totalImagesPullingTime="11.740486608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:12.881372118 +0000 UTC m=+23.458733270" watchObservedRunningTime="2026-09-16 14:08:12.881807013 +0000 UTC m=+23.459168161" Sep 16 14:08:12.882597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"23d0b0d7630df5bff55719a0a9b7f8d758f7ccef055a0e55570729b9614ffa95"} Sep 16 14:08:12.882714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"f81aa59f411c69f825249aadef739a632472b8d3f5a04427934b2772d5a4de32"} Sep 16 14:08:12.882714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"8c0ca0c70b687d38f742119fe67e30e798885ec5b7da7c30494c501b464bfcf7"} Sep 16 14:08:12.882820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"dca4c8fae9019583f7938007b3e50de907b381393a11513a86cd812072b8b89a"} Sep 16 14:08:12.882820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"caf7e93c5b4ebc2a84c381d74bb05c0e4ce9447f80c7bc188c56e844e44b7b8b"} Sep 16 14:08:12.882820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.882804 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"696078dce18380a7e311a9725ad03717c195b840f35a91b82893c886dbcab4df"} Sep 16 14:08:12.884526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.884282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" event={"ID":"579298bbec0b9e8e74fae2628d563376","Type":"ContainerStarted","Data":"2746f7df98627e03e474c9610b24f8099bbc46aa4dbf941cc8fc5a99e09a5523"} Sep 16 14:08:12.886349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.886319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4rqpt" event={"ID":"87d538d9-4506-4269-8ec6-5d4d1c8e402a","Type":"ContainerStarted","Data":"6c01708fd376d0a137aca9d119cca541b1bf1683292b134f542bf4f96befebb9"} Sep 16 14:08:12.887980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.887953 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="30b17f4b7bfb127f1e18122c21f35983aabdb5987db2c9b2a8d84c393441576e" exitCode=0 Sep 16 14:08:12.888069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.887987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"30b17f4b7bfb127f1e18122c21f35983aabdb5987db2c9b2a8d84c393441576e"} Sep 16 14:08:12.931700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.931663 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zsbg2" podStartSLOduration=11.180785628 podStartE2EDuration="22.931647533s" totalImagesPullingTime="11.750861905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:12.931645602 +0000 UTC m=+23.509006775" watchObservedRunningTime="2026-09-16 14:08:12.931647533 +0000 UTC m=+23.509008687" Sep 16 14:08:12.936175 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.936132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l8mng" podStartSLOduration=3.071396617 podStartE2EDuration="22.93611539s" totalImagesPullingTime="19.864718773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:12.914273657 +0000 UTC m=+23.491634810" watchObservedRunningTime="2026-09-16 14:08:12.93611539 +0000 UTC m=+23.513476543" Sep 16 14:08:12.947784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:12.947740 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b97t4" podStartSLOduration=3.166506336 podStartE2EDuration="22.947723373s" totalImagesPullingTime="19.781217037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:12.947145479 +0000 UTC m=+23.524506633" watchObservedRunningTime="2026-09-16 14:08:12.947723373 +0000 UTC m=+23.525084526" Sep 16 14:08:13.015326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.015286 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4rqpt" podStartSLOduration=11.366840661 podStartE2EDuration="22.015268714s" totalImagesPullingTime="10.648428053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:13.015229398 +0000 UTC m=+23.592590549" watchObservedRunningTime="2026-09-16 14:08:13.015268714 +0000 UTC m=+23.592629867" Sep 16 14:08:13.030636 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.030565 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-170.ec2.internal" podStartSLOduration=21.03054699 podStartE2EDuration="21.03054699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:13.029951572 +0000 UTC m=+23.607312724" watchObservedRunningTime="2026-09-16 14:08:13.03054699 +0000 UTC m=+23.607908143" Sep 16 14:08:13.317145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.316989 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 14:08:13.810761 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.810731 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:13.810761 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.810760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:13.810986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:13.810861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:13.810986 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:13.810907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:13.810986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.810960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:13.811109 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:13.811025 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:13.818749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.818654 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T14:08:13.317136102Z","UUID":"05323a6a-1d90-47dc-bf20-c25354019533","Handler":null,"Name":"","Endpoint":""} Sep 16 14:08:13.820319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.820303 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 14:08:13.820406 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.820332 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 14:08:13.892864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.892832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" event={"ID":"bf7554308d9b300ebecd820a92d9c922","Type":"ContainerStarted","Data":"d2cc5a94e8ad55014268c6d2ae5a0982e3f60ea276ff99c771bf0ffeb60a0beb"} Sep 16 14:08:13.894998 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.894968 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" event={"ID":"6ae28b05-e749-47b2-afb3-51fee098588b","Type":"ContainerStarted","Data":"12530b18a73171d30c79c2b21161a95f75bcfd38424b8087c286f2861bc80d1b"} Sep 16 14:08:13.896756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.896725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28htz" event={"ID":"ae4ab7a4-b522-4b58-8b04-3379f14e2b3e","Type":"ContainerStarted","Data":"288e5449db341b064e2fb1aea4a1797d0140c635f2511e361d3e2e48ace874c9"} Sep 16 14:08:13.909454 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.909405 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-170.ec2.internal" podStartSLOduration=21.909387817 podStartE2EDuration="21.909387817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:13.908436786 +0000 UTC m=+24.485797940" watchObservedRunningTime="2026-09-16 14:08:13.909387817 +0000 UTC m=+24.486748970" Sep 16 14:08:13.925920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:13.925873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-28htz" podStartSLOduration=12.156569082 podStartE2EDuration="23.925861294s" totalImagesPullingTime="11.769292212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:13.925404044 +0000 UTC m=+24.502765195" watchObservedRunningTime="2026-09-16 14:08:13.925861294 +0000 UTC m=+24.503222446" Sep 16 14:08:14.902255 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:14.902159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" event={"ID":"6ae28b05-e749-47b2-afb3-51fee098588b","Type":"ContainerStarted","Data":"d2d7c23a788feee6bbb626471d8040111b510b9aea142917fa513c83b7909ba2"} Sep 16 14:08:14.905984 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:14.905948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"bc2665e14aa176b12c8818eb5e8ca4c2998e8cedad4cc4a52f6a2376139cd309"} Sep 16 14:08:14.923133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:14.923083 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4k5c" podStartSLOduration=2.076961164 podStartE2EDuration="23.923067949s" totalImagesPullingTime="21.846106785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:14.922545161 +0000 UTC m=+25.499906309" watchObservedRunningTime="2026-09-16 14:08:14.923067949 +0000 UTC m=+25.500429101" Sep 16 14:08:15.810675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:15.810633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:15.810887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:15.810630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:15.810887 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:15.810763 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:15.810887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:15.810643 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:15.811055 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:15.810828 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:15.811055 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:15.810908 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:17.811205 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.811176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:17.811205 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.811209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:17.811643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.811175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:17.811643 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:17.811295 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:17.811643 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:17.811394 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:17.811643 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:17.811443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:17.839020 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.838998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:08:17.839507 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.839489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:08:17.912967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.912943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:08:17.913396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:17.913378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k76cg" Sep 16 14:08:18.918867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.918536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27422" event={"ID":"95fc6c93-c45e-4f95-9e0c-8f884706637b","Type":"ContainerStarted","Data":"107cc484de7d3c0eabdc95bef8d97b8ea4c1f453ac19947e41e3483d34426753"} Sep 16 14:08:18.919325 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.918871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:18.920722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.920696 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="5e93ef9ab52277ce7a640093accd743e73c13b7d70e47994f4669f9c4d2df761" exitCode=0 Sep 16 14:08:18.920820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.920786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"5e93ef9ab52277ce7a640093accd743e73c13b7d70e47994f4669f9c4d2df761"} Sep 16 14:08:18.935071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.935051 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:18.948467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:18.948430 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-27422" podStartSLOduration=9.108710914 podStartE2EDuration="28.948417038s" totalImagesPullingTime="19.839706124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:18.948245745 +0000 UTC m=+29.525606898" watchObservedRunningTime="2026-09-16 14:08:18.948417038 +0000 UTC m=+29.525778190" Sep 16 14:08:19.812199 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.812169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:19.812324 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:19.812288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:19.812414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.812389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:19.812582 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:19.812556 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:19.812670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.812582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:19.812709 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:19.812682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:19.924434 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.923336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:19.924434 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.923446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:19.944863 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:19.944842 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:20.043834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.043807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbr5h"] Sep 16 14:08:20.043937 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.043900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:20.043995 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:20.043977 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:20.045580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.045559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cgs2"] Sep 16 14:08:20.045687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.045652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:20.045735 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:20.045719 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:20.051314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.051291 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lx7g"] Sep 16 14:08:20.051382 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.051370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:20.051501 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:20.051476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:20.927319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.927285 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="8313792848ec5e9cbc07bcffcc660ac7bc82da2d9f603ff17f37f895b38a8d44" exitCode=0 Sep 16 14:08:20.927777 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:20.927367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"8313792848ec5e9cbc07bcffcc660ac7bc82da2d9f603ff17f37f895b38a8d44"} Sep 16 14:08:21.810723 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:21.810459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:21.810906 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:21.810485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:21.810906 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:21.810816 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:21.810906 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:21.810491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:21.811065 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:21.810905 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:21.811065 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:21.810934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:23.611783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.611691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:23.611783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.611734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:23.612191 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.611854 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 14:08:23.612191 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.611919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret podName:bb787601-e0a7-43ca-a6c9-6f66b529995b nodeName:}" failed. No retries permitted until 2026-09-16 14:08:55.611905566 +0000 UTC m=+66.189266695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret") pod "global-pull-secret-syncer-rbr5h" (UID: "bb787601-e0a7-43ca-a6c9-6f66b529995b") : object "kube-system"/"original-pull-secret" not registered Sep 16 14:08:23.612191 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.611853 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:08:23.612191 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.611989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs podName:01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:55.611978487 +0000 UTC m=+66.189339616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs") pod "network-metrics-daemon-7lx7g" (UID: "01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 14:08:23.712598 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.712560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:23.712778 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.712725 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 14:08:23.712778 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.712745 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 14:08:23.712778 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.712754 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hvf2n for pod openshift-network-diagnostics/network-check-target-5cgs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:08:23.712871 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.712807 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n podName:573364cf-e95e-4380-9e10-8c0265047660 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:55.712791531 +0000 UTC m=+66.290152661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvf2n" (UniqueName: "kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n") pod "network-check-target-5cgs2" (UID: "573364cf-e95e-4380-9e10-8c0265047660") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 14:08:23.810478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.810441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:23.810678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.810446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:23.810678 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.810553 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cgs2" podUID="573364cf-e95e-4380-9e10-8c0265047660" Sep 16 14:08:23.810777 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.810446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:23.810777 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.810681 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbr5h" podUID="bb787601-e0a7-43ca-a6c9-6f66b529995b" Sep 16 14:08:23.810845 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:23.810779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx7g" podUID="01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1" Sep 16 14:08:23.935879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.935846 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="3e12ec08fd46d4a8435f4a25a28db13e2ad3c5e317bdb53f32fc2f7ed4c73cad" exitCode=0 Sep 16 14:08:23.936024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:23.935927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"3e12ec08fd46d4a8435f4a25a28db13e2ad3c5e317bdb53f32fc2f7ed4c73cad"} Sep 16 14:08:24.759376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.759344 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-170.ec2.internal" event="NodeReady" Sep 16 14:08:24.759834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.759496 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 14:08:24.813838 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.813806 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd"] Sep 16 14:08:24.829238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.829207 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:08:24.829390 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.829332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:24.831756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.831725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-96brr\"" Sep 16 14:08:24.832055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.832030 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.832151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.832105 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.832216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.832032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 14:08:24.857198 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.857173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c"] Sep 16 14:08:24.857324 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.857308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:24.862318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.861828 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 14:08:24.862318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.861996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 14:08:24.862318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.862147 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ffjnt\"" Sep 16 14:08:24.862318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.862270 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 14:08:24.871968 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.871945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 14:08:24.873238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.873220 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-685877fd7d-tp56g"] Sep 16 14:08:24.873381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.873365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:24.879755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.879732 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 14:08:24.879988 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.879969 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.880450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.880431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wlnh4\"" Sep 16 14:08:24.880536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.880501 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.880536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.880520 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 14:08:24.895466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.895440 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw"] Sep 16 14:08:24.895663 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.895580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:24.899234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.899213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 14:08:24.899919 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.899900 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.900983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.900517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-82498\"" Sep 16 14:08:24.900983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.900519 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.900983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.900836 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 14:08:24.901289 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.901269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 14:08:24.901444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.901299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 14:08:24.913410 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.913388 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz"] Sep 16 14:08:24.913518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.913494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:24.916659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.916639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 14:08:24.917451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.916962 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 14:08:24.917451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.917048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9n5lm\"" Sep 16 14:08:24.917451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.917154 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.917451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.917325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.920117 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.920094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:24.920234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.920164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdddx\" (UniqueName: \"kubernetes.io/projected/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-kube-api-access-pdddx\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:24.937488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.937436 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq"] Sep 16 14:08:24.937592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.937546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" Sep 16 14:08:24.940508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.940487 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cjxbj\"" Sep 16 14:08:24.940661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.940490 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.940661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.940544 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.952188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.952170 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llzzn"] Sep 16 14:08:24.952308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.952292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:24.956643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.956581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:24.956867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.956844 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.956951 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.956914 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 14:08:24.957080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.957044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 14:08:24.957080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.957077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.957188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.956844 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nrxcf\"" Sep 16 14:08:24.970323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.970303 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd"] Sep 16 14:08:24.970425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.970348 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-z6mkc"] Sep 16 14:08:24.970488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.970430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:24.972911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.972784 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 14:08:24.972911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.972806 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.972911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.972807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qvfpp\"" Sep 16 14:08:24.973185 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.973170 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 14:08:24.973497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.973482 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 14:08:24.980755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.980729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 14:08:24.990452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:24.990452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990448 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990517 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-685877fd7d-tp56g"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llzzn"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-z6mkc"] Sep 16 14:08:24.990623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.990585 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw"] Sep 16 14:08:24.994168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.994150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 14:08:24.994644 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.994627 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 14:08:24.994751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.994629 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 14:08:24.994994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.994980 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dv6rq\"" Sep 16 14:08:24.995462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:24.995445 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 14:08:25.000890 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.000871 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 14:08:25.004548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.004530 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-66tjl"] Sep 16 14:08:25.004690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.004674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" Sep 16 14:08:25.011435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.011415 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 14:08:25.011435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.011416 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 14:08:25.011576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.011498 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bfq47\"" Sep 16 14:08:25.018733 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.018717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qmqk4"] Sep 16 14:08:25.018843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.018827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.021034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbkd\" (UniqueName: \"kubernetes.io/projected/03e3a050-8164-4248-b342-549e1ebaa9f8-kube-api-access-5hbkd\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.021140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe825bb-5841-464d-9dd4-1b41c72c77a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.021140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.021140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.021140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/efe825bb-5841-464d-9dd4-1b41c72c77a9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkr5\" (UniqueName: \"kubernetes.io/projected/efe825bb-5841-464d-9dd4-1b41c72c77a9-kube-api-access-5vkr5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03e3a050-8164-4248-b342-549e1ebaa9f8-serving-cert\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdddx\" (UniqueName: \"kubernetes.io/projected/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-kube-api-access-pdddx\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-stats-auth\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2xp\" (UniqueName: \"kubernetes.io/projected/8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1-kube-api-access-gw2xp\") pod \"volume-data-source-validator-86695b7cc7-sd6jz\" (UID: \"8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021507 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 14:08:25.021553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mx8\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021565 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-default-certificate\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021702 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbr4\" (UniqueName: \"kubernetes.io/projected/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-kube-api-access-npbr4\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03e3a050-8164-4248-b342-549e1ebaa9f8-config\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8lnxj\"" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.021770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.021878 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:25.022045 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.021931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.521918331 +0000 UTC m=+36.099279474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:25.032703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.032685 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq"] Sep 16 14:08:25.032810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.032713 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw"] Sep 16 14:08:25.032810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.032727 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmqk4"] Sep 16 14:08:25.032810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.032742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66tjl"] Sep 16 14:08:25.032810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.032784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.034065 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.034031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdddx\" (UniqueName: \"kubernetes.io/projected/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-kube-api-access-pdddx\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:25.035344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.035319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 14:08:25.035494 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.035428 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 14:08:25.035651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.035634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8lck4\"" Sep 16 14:08:25.064211 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.064171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:25.122960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.122929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-default-certificate\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.122960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.122969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.123217 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.122997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.123217 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.123099 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:25.123217 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.123159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.62313911 +0000 UTC m=+36.200500240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:25.123378 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-trusted-ca\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.123378 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-service-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.123378 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-tmp\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.123565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-snapshots\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.123565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.123565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhbj\" (UniqueName: \"kubernetes.io/projected/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-kube-api-access-qkhbj\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.123565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbm7\" (UniqueName: \"kubernetes.io/projected/85cf4dad-2643-4990-8be7-be90f1faa7c8-kube-api-access-pvbm7\") pod \"network-check-source-6f8b768455-n8jzw\" (UID: \"85cf4dad-2643-4990-8be7-be90f1faa7c8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmz4\" (UniqueName: \"kubernetes.io/projected/2cce6995-91cf-4a57-87ca-ee871b0751b8-kube-api-access-nqmz4\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-config-volume\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-tmp-dir\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/efe825bb-5841-464d-9dd4-1b41c72c77a9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.123788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbj9x\" (UniqueName: \"kubernetes.io/projected/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-kube-api-access-jbj9x\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-config\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.123843 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.623824897 +0000 UTC m=+36.201186055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-serving-cert\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-stats-auth\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac63b38a-300e-483e-94b4-06db8633505f-serving-cert\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.124151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.123987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdbs\" (UniqueName: \"kubernetes.io/projected/ac63b38a-300e-483e-94b4-06db8633505f-kube-api-access-gkdbs\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2xp\" (UniqueName: \"kubernetes.io/projected/8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1-kube-api-access-gw2xp\") pod \"volume-data-source-validator-86695b7cc7-sd6jz\" (UID: \"8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbkd\" (UniqueName: \"kubernetes.io/projected/03e3a050-8164-4248-b342-549e1ebaa9f8-kube-api-access-5hbkd\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mx8\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbr4\" (UniqueName: \"kubernetes.io/projected/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-kube-api-access-npbr4\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2hl\" (UniqueName: \"kubernetes.io/projected/574a4a71-86e6-4ed6-9080-404c193e5884-kube-api-access-jw2hl\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cce6995-91cf-4a57-87ca-ee871b0751b8-cert\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03e3a050-8164-4248-b342-549e1ebaa9f8-config\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe825bb-5841-464d-9dd4-1b41c72c77a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.124487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/efe825bb-5841-464d-9dd4-1b41c72c77a9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/574a4a71-86e6-4ed6-9080-404c193e5884-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkr5\" (UniqueName: \"kubernetes.io/projected/efe825bb-5841-464d-9dd4-1b41c72c77a9-kube-api-access-5vkr5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03e3a050-8164-4248-b342-549e1ebaa9f8-serving-cert\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.125012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.124696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.125299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.125090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.125299 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.125183 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:25.125299 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.125197 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:25.125299 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.125238 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.625222968 +0000 UTC m=+36.202584104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:25.125799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.125512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.125799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.125707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.126499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.126273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03e3a050-8164-4248-b342-549e1ebaa9f8-config\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.127515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.127492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-stats-auth\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.127908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.127860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.128253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.128234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03e3a050-8164-4248-b342-549e1ebaa9f8-serving-cert\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.128767 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.128747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe825bb-5841-464d-9dd4-1b41c72c77a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.130243 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.130225 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-default-certificate\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.131836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.131810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.135450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.135426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbkd\" (UniqueName: \"kubernetes.io/projected/03e3a050-8164-4248-b342-549e1ebaa9f8-kube-api-access-5hbkd\") pod \"service-ca-operator-cf7c776b7-5nlgw\" (UID: \"03e3a050-8164-4248-b342-549e1ebaa9f8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.135582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.135556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.136277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.136236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbr4\" (UniqueName: \"kubernetes.io/projected/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-kube-api-access-npbr4\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.136585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.136520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkr5\" (UniqueName: \"kubernetes.io/projected/efe825bb-5841-464d-9dd4-1b41c72c77a9-kube-api-access-5vkr5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-skt6c\" (UID: \"efe825bb-5841-464d-9dd4-1b41c72c77a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.136973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.136953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mx8\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.137308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.137291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2xp\" (UniqueName: \"kubernetes.io/projected/8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1-kube-api-access-gw2xp\") pod \"volume-data-source-validator-86695b7cc7-sd6jz\" (UID: \"8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" Sep 16 14:08:25.183047 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.183009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" Sep 16 14:08:25.224399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.224366 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" Sep 16 14:08:25.225598 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cce6995-91cf-4a57-87ca-ee871b0751b8-cert\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.225709 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/574a4a71-86e6-4ed6-9080-404c193e5884-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.225771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-trusted-ca\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.225771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-service-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.225873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-tmp\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.225873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-snapshots\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.225873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.225873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhbj\" (UniqueName: \"kubernetes.io/projected/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-kube-api-access-qkhbj\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.226064 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.226012 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:25.226116 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.226075 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.726054917 +0000 UTC m=+36.303416063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-tmp\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/574a4a71-86e6-4ed6-9080-404c193e5884-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-service-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac63b38a-300e-483e-94b4-06db8633505f-snapshots\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.225877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbm7\" (UniqueName: \"kubernetes.io/projected/85cf4dad-2643-4990-8be7-be90f1faa7c8-kube-api-access-pvbm7\") pod \"network-check-source-6f8b768455-n8jzw\" (UID: \"85cf4dad-2643-4990-8be7-be90f1faa7c8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmz4\" (UniqueName: \"kubernetes.io/projected/2cce6995-91cf-4a57-87ca-ee871b0751b8-kube-api-access-nqmz4\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-config-volume\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-tmp-dir\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbj9x\" (UniqueName: \"kubernetes.io/projected/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-kube-api-access-jbj9x\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.226982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-config\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-serving-cert\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac63b38a-300e-483e-94b4-06db8633505f-serving-cert\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdbs\" (UniqueName: \"kubernetes.io/projected/ac63b38a-300e-483e-94b4-06db8633505f-kube-api-access-gkdbs\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.227352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2hl\" (UniqueName: \"kubernetes.io/projected/574a4a71-86e6-4ed6-9080-404c193e5884-kube-api-access-jw2hl\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-trusted-ca\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.227386 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.227433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:25.72741804 +0000 UTC m=+36.304779172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-config-volume\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.227578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac63b38a-300e-483e-94b4-06db8633505f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.228509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.228025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-config\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.229058 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.229035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-tmp-dir\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.229518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.229470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cce6995-91cf-4a57-87ca-ee871b0751b8-cert\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.230141 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.230121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-serving-cert\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.231406 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.231387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac63b38a-300e-483e-94b4-06db8633505f-serving-cert\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.239799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.239774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbj9x\" (UniqueName: \"kubernetes.io/projected/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-kube-api-access-jbj9x\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.239990 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.239969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhbj\" (UniqueName: \"kubernetes.io/projected/d9f92411-315d-4c2e-ab0c-22332bf4c8c7-kube-api-access-qkhbj\") pod \"console-operator-696884fb9c-llzzn\" (UID: \"d9f92411-315d-4c2e-ab0c-22332bf4c8c7\") " pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.240306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.240286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbm7\" (UniqueName: \"kubernetes.io/projected/85cf4dad-2643-4990-8be7-be90f1faa7c8-kube-api-access-pvbm7\") pod \"network-check-source-6f8b768455-n8jzw\" (UID: \"85cf4dad-2643-4990-8be7-be90f1faa7c8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" Sep 16 14:08:25.241309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.241284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2hl\" (UniqueName: \"kubernetes.io/projected/574a4a71-86e6-4ed6-9080-404c193e5884-kube-api-access-jw2hl\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.241873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.241854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdbs\" (UniqueName: \"kubernetes.io/projected/ac63b38a-300e-483e-94b4-06db8633505f-kube-api-access-gkdbs\") pod \"insights-operator-5844b856db-z6mkc\" (UID: \"ac63b38a-300e-483e-94b4-06db8633505f\") " pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.241985 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.241962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmz4\" (UniqueName: \"kubernetes.io/projected/2cce6995-91cf-4a57-87ca-ee871b0751b8-kube-api-access-nqmz4\") pod \"ingress-canary-66tjl\" (UID: \"2cce6995-91cf-4a57-87ca-ee871b0751b8\") " pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.260701 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.260679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" Sep 16 14:08:25.280070 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.280032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:25.302778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.301931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-z6mkc" Sep 16 14:08:25.312285 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.312196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" Sep 16 14:08:25.330624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.330468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66tjl" Sep 16 14:08:25.517043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.517012 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c"] Sep 16 14:08:25.521997 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.521965 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe825bb_5841_464d_9dd4_1b41c72c77a9.slice/crio-c2079fdc7b1580146e85e06dee6d388ee9cfbff9edbb17fbb786f695a220bc8a WatchSource:0}: Error finding container c2079fdc7b1580146e85e06dee6d388ee9cfbff9edbb17fbb786f695a220bc8a: Status 404 returned error can't find the container with id c2079fdc7b1580146e85e06dee6d388ee9cfbff9edbb17fbb786f695a220bc8a Sep 16 14:08:25.530114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.529653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:25.530114 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.529909 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:25.530114 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.529965 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.529946902 +0000 UTC m=+37.107308038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:25.530975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.530952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw"] Sep 16 14:08:25.534021 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.533998 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e3a050_8164_4248_b342_549e1ebaa9f8.slice/crio-96b50b4fe1ff182c8ac656aa6335339599ff2bb7dda99d447a12b578b7ba5d64 WatchSource:0}: Error finding container 96b50b4fe1ff182c8ac656aa6335339599ff2bb7dda99d447a12b578b7ba5d64: Status 404 returned error can't find the container with id 96b50b4fe1ff182c8ac656aa6335339599ff2bb7dda99d447a12b578b7ba5d64 Sep 16 14:08:25.597031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.597007 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz"] Sep 16 14:08:25.599420 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.599391 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8350b6d8_cdf2_4816_9ca1_b16f6bc0f5d1.slice/crio-abd00052d23ef3a794f8839c470992fb5033281f375e26934beedbc15860680a WatchSource:0}: Error finding container abd00052d23ef3a794f8839c470992fb5033281f375e26934beedbc15860680a: Status 404 returned error can't find the container with id abd00052d23ef3a794f8839c470992fb5033281f375e26934beedbc15860680a Sep 16 14:08:25.623792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.623771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llzzn"] Sep 16 14:08:25.626211 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.626188 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f92411_315d_4c2e_ab0c_22332bf4c8c7.slice/crio-680c268221649313cf33d369f46ccef31e274bffc264533175162736ccf184d1 WatchSource:0}: Error finding container 680c268221649313cf33d369f46ccef31e274bffc264533175162736ccf184d1: Status 404 returned error can't find the container with id 680c268221649313cf33d369f46ccef31e274bffc264533175162736ccf184d1 Sep 16 14:08:25.630685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.630667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.630788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.630772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:25.630839 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.630825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.630808192 +0000 UTC m=+37.208169341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:25.630875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.630852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:25.630908 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.630880 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:25.630908 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.630892 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:25.630962 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.630940 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.630916446 +0000 UTC m=+37.208277578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:25.630994 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.630973 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:25.631026 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.631009 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.63100114 +0000 UTC m=+37.208362270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:25.647257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.647238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-z6mkc"] Sep 16 14:08:25.649563 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.649536 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac63b38a_300e_483e_94b4_06db8633505f.slice/crio-14e86e2fbb39f180c19feccb07b6cfdb60a80e11b2528a69f493fe83871ccb38 WatchSource:0}: Error finding container 14e86e2fbb39f180c19feccb07b6cfdb60a80e11b2528a69f493fe83871ccb38: Status 404 returned error can't find the container with id 14e86e2fbb39f180c19feccb07b6cfdb60a80e11b2528a69f493fe83871ccb38 Sep 16 14:08:25.665664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.665639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66tjl"] Sep 16 14:08:25.667186 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.667155 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cce6995_91cf_4a57_87ca_ee871b0751b8.slice/crio-a5e809c040c30066f66756c0c6e1b0bf998d3bd7feac125352a357c620ab0cb1 WatchSource:0}: Error finding container a5e809c040c30066f66756c0c6e1b0bf998d3bd7feac125352a357c620ab0cb1: Status 404 returned error can't find the container with id a5e809c040c30066f66756c0c6e1b0bf998d3bd7feac125352a357c620ab0cb1 Sep 16 14:08:25.694348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.694319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw"] Sep 16 14:08:25.697056 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:25.697035 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cf4dad_2643_4990_8be7_be90f1faa7c8.slice/crio-ce41f03e5eba019849cf23413351eb967ccc47f5925ddd94b33c753d8e399e7c WatchSource:0}: Error finding container ce41f03e5eba019849cf23413351eb967ccc47f5925ddd94b33c753d8e399e7c: Status 404 returned error can't find the container with id ce41f03e5eba019849cf23413351eb967ccc47f5925ddd94b33c753d8e399e7c Sep 16 14:08:25.731676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.731600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:25.731785 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.731691 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:25.731785 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.731708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:25.731785 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.731741 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.731727597 +0000 UTC m=+37.309088732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:25.731930 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.731802 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:25.731930 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:25.731847 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:26.731832139 +0000 UTC m=+37.309193272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:25.811192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.811159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:25.811632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.811159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:25.811632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.811170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:25.814505 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.814476 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8mpwg\"" Sep 16 14:08:25.814871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.814849 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 14:08:25.815009 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.814995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 14:08:25.815078 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.815063 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x4gh8\"" Sep 16 14:08:25.942994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.942950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" event={"ID":"d9f92411-315d-4c2e-ab0c-22332bf4c8c7","Type":"ContainerStarted","Data":"680c268221649313cf33d369f46ccef31e274bffc264533175162736ccf184d1"} Sep 16 14:08:25.944262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.944231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" event={"ID":"8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1","Type":"ContainerStarted","Data":"abd00052d23ef3a794f8839c470992fb5033281f375e26934beedbc15860680a"} Sep 16 14:08:25.945497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.945472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" event={"ID":"03e3a050-8164-4248-b342-549e1ebaa9f8","Type":"ContainerStarted","Data":"96b50b4fe1ff182c8ac656aa6335339599ff2bb7dda99d447a12b578b7ba5d64"} Sep 16 14:08:25.946666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.946639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-z6mkc" event={"ID":"ac63b38a-300e-483e-94b4-06db8633505f","Type":"ContainerStarted","Data":"14e86e2fbb39f180c19feccb07b6cfdb60a80e11b2528a69f493fe83871ccb38"} Sep 16 14:08:25.948401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.948373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66tjl" event={"ID":"2cce6995-91cf-4a57-87ca-ee871b0751b8","Type":"ContainerStarted","Data":"a5e809c040c30066f66756c0c6e1b0bf998d3bd7feac125352a357c620ab0cb1"} Sep 16 14:08:25.949650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.949627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" event={"ID":"efe825bb-5841-464d-9dd4-1b41c72c77a9","Type":"ContainerStarted","Data":"c2079fdc7b1580146e85e06dee6d388ee9cfbff9edbb17fbb786f695a220bc8a"} Sep 16 14:08:25.950962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:25.950916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" event={"ID":"85cf4dad-2643-4990-8be7-be90f1faa7c8","Type":"ContainerStarted","Data":"ce41f03e5eba019849cf23413351eb967ccc47f5925ddd94b33c753d8e399e7c"} Sep 16 14:08:26.540174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.540135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:26.540418 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.540400 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:26.540477 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.540471 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.540450877 +0000 UTC m=+39.117812012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:26.641068 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.641032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:26.641222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.641098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:26.641222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.641172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:26.641350 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.64130372 +0000 UTC m=+39.218664870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:26.641896 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641765 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:26.641896 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641785 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:26.641896 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641805 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:26.641896 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.641814399 +0000 UTC m=+39.219175545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:26.641896 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.641846 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.641835088 +0000 UTC m=+39.219196219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:26.742714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.742622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:26.742885 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:26.742729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:26.742885 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.742875 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:26.743001 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.742936 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.742918708 +0000 UTC m=+39.320279843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:26.743663 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.743282 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:26.743663 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:26.743323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:28.743313129 +0000 UTC m=+39.320674258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:28.560991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.560957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:28.561848 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.561133 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:28.561848 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.561212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.561191863 +0000 UTC m=+43.138553023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:28.661917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.661878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:28.662108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.661955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:28.662108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.662016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:28.662108 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662109 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:28.662284 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662119 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:28.662284 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662163 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.662150652 +0000 UTC m=+43.239511795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:28.662459 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662448 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:28.662520 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662485 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.662475388 +0000 UTC m=+43.239836531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:28.662584 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.662531 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.662525281 +0000 UTC m=+43.239886424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:28.763248 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.763215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:28.763428 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.763341 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:28.763428 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:28.763352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:28.763428 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.763402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.763387441 +0000 UTC m=+43.340748570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:28.763582 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.763486 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:28.763582 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:28.763546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:32.763530554 +0000 UTC m=+43.340891686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:30.554192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:30.554116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:30.654661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:30.654619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:32.597660 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.597624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:32.598086 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.597774 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:32.598086 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.597848 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.597827987 +0000 UTC m=+51.175189121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:32.699108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.699070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:32.699296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.699161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:32.699296 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699234 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:32.699296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.699259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:32.699418 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699296 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.699280958 +0000 UTC m=+51.276642092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:32.699418 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699313 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.699307346 +0000 UTC m=+51.276668475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:32.699495 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699413 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:32.699495 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699432 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:32.699495 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.699474 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.699461712 +0000 UTC m=+51.276822842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:32.800186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.800145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:32.800387 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:32.800243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:32.800387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.800297 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:32.800387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.800372 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.800355869 +0000 UTC m=+51.377717022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:32.800387 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.800373 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:32.800598 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:32.800422 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.800410892 +0000 UTC m=+51.377772022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:35.641282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.641252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:35.750715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.750030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:35.912808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.912770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:35.988024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.987992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" event={"ID":"85cf4dad-2643-4990-8be7-be90f1faa7c8","Type":"ContainerStarted","Data":"9e3dbc1e9e9963dbf2636477c5a65d6bb6237ba60cb4b23f05775b014e9e81b1"} Sep 16 14:08:35.990149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.990128 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/0.log" Sep 16 14:08:35.990253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.990170 2579 generic.go:356] "Generic (PLEG): container finished" podID="d9f92411-315d-4c2e-ab0c-22332bf4c8c7" containerID="2ee0b3792948da4a27de9828d86e037fc00f1570ea2ff7f970f946dc6ad55ce7" exitCode=255 Sep 16 14:08:35.990253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.990246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" event={"ID":"d9f92411-315d-4c2e-ab0c-22332bf4c8c7","Type":"ContainerDied","Data":"2ee0b3792948da4a27de9828d86e037fc00f1570ea2ff7f970f946dc6ad55ce7"} Sep 16 14:08:35.990714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.990411 2579 scope.go:117] "RemoveContainer" containerID="2ee0b3792948da4a27de9828d86e037fc00f1570ea2ff7f970f946dc6ad55ce7" Sep 16 14:08:35.993132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.992635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" event={"ID":"8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1","Type":"ContainerStarted","Data":"acb3209db372e02885303bc676d52aad4c752905d2137ef2549a18b52871e608"} Sep 16 14:08:35.994959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.994932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" event={"ID":"03e3a050-8164-4248-b342-549e1ebaa9f8","Type":"ContainerStarted","Data":"2e5280e75eca9b08d66592e27887cbff7a6dde97e8cf9cf263c674b5c4bc558c"} Sep 16 14:08:35.996557 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.996534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-z6mkc" event={"ID":"ac63b38a-300e-483e-94b4-06db8633505f","Type":"ContainerStarted","Data":"02f3ea42168701d28d8260bb1469e4d1f4c00b5fdfc5cd5e39d1edaac47c2a78"} Sep 16 14:08:35.999941 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.999910 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="0375033b3db709dffc2e0836c91d6eba50584b044b1042d7258dfac03b66489a" exitCode=0 Sep 16 14:08:36.000037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:35.999991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"0375033b3db709dffc2e0836c91d6eba50584b044b1042d7258dfac03b66489a"} Sep 16 14:08:36.002788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.002768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66tjl" event={"ID":"2cce6995-91cf-4a57-87ca-ee871b0751b8","Type":"ContainerStarted","Data":"5d0f45854f6bda71afb3c936fd1cefef4a1ac463b8165119cf2b96491971fdf0"} Sep 16 14:08:36.005730 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.004705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" event={"ID":"efe825bb-5841-464d-9dd4-1b41c72c77a9","Type":"ContainerStarted","Data":"6f50b59a4efee24cf56734878cef6d515369452e91601a96caf9ac5e133ea12b"} Sep 16 14:08:36.015247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.015203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-n8jzw" podStartSLOduration=13.18847574 podStartE2EDuration="23.015186152s" totalImagesPullingTime="9.826710412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.014153606 +0000 UTC m=+46.591514760" watchObservedRunningTime="2026-09-16 14:08:36.015186152 +0000 UTC m=+46.592547299" Sep 16 14:08:36.020477 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.020364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.039928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.039882 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" podStartSLOduration=13.06277173 podStartE2EDuration="23.039863653s" totalImagesPullingTime="9.977091923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.039543138 +0000 UTC m=+46.616904291" watchObservedRunningTime="2026-09-16 14:08:36.039863653 +0000 UTC m=+46.617224808" Sep 16 14:08:36.068541 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.068497 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" podStartSLOduration=13.088399144 podStartE2EDuration="23.068475334s" totalImagesPullingTime="9.98007619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.067101792 +0000 UTC m=+46.644462944" watchObservedRunningTime="2026-09-16 14:08:36.068475334 +0000 UTC m=+46.645836466" Sep 16 14:08:36.114210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.114159 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-66tjl" podStartSLOduration=2.266968602 podStartE2EDuration="12.114136597s" totalImagesPullingTime="9.847167995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.092081249 +0000 UTC m=+46.669442402" watchObservedRunningTime="2026-09-16 14:08:36.114136597 +0000 UTC m=+46.691497750" Sep 16 14:08:36.114630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.114565 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-sd6jz" podStartSLOduration=13.28328923 podStartE2EDuration="23.11455498s" totalImagesPullingTime="9.83126575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.113887045 +0000 UTC m=+46.691248198" watchObservedRunningTime="2026-09-16 14:08:36.11455498 +0000 UTC m=+46.691916134" Sep 16 14:08:36.132859 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.132787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.164357 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.164289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-z6mkc" podStartSLOduration=13.29956737 podStartE2EDuration="23.164266564s" totalImagesPullingTime="9.864699194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:36.163406749 +0000 UTC m=+46.740767902" watchObservedRunningTime="2026-09-16 14:08:36.164266564 +0000 UTC m=+46.741627711" Sep 16 14:08:36.537396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.537352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.620042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.620006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.768681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.768643 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4sqqr"] Sep 16 14:08:36.772694 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.772664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.784718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.784688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.788409 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.788356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 14:08:36.789411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.789396 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 14:08:36.789836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.789815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c7jvw\"" Sep 16 14:08:36.791052 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.791031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4sqqr"] Sep 16 14:08:36.838064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.838032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-data-volume\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.838219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.838076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-crio-socket\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.838219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.838106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gzk\" (UniqueName: \"kubernetes.io/projected/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-api-access-c8gzk\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.838219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.838146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.838219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.838181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.846638 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.846589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:36.938908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.938871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-crio-socket\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.938908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.938910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gzk\" (UniqueName: \"kubernetes.io/projected/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-api-access-c8gzk\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.939101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.938931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.939101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.938960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.939101 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:36.939042 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 14:08:36.939101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.939063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-crio-socket\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.939295 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:36.939114 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:37.439097128 +0000 UTC m=+48.016458280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : configmap "kube-rbac-proxy" not found Sep 16 14:08:36.939295 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:36.939126 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 14:08:36.939295 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:36.939184 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:37.43916987 +0000 UTC m=+48.016531000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : secret "insights-runtime-extractor-tls" not found Sep 16 14:08:36.939295 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.939226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-data-volume\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.939506 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.939492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-data-volume\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:36.952332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:36.952308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gzk\" (UniqueName: \"kubernetes.io/projected/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-api-access-c8gzk\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:37.012572 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.012540 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:08:37.014074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014051 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/0.log" Sep 16 14:08:37.014208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014093 2579 generic.go:356] "Generic (PLEG): container finished" podID="d9f92411-315d-4c2e-ab0c-22332bf4c8c7" containerID="55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76" exitCode=255 Sep 16 14:08:37.014208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.014311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" event={"ID":"d9f92411-315d-4c2e-ab0c-22332bf4c8c7","Type":"ContainerDied","Data":"55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76"} Sep 16 14:08:37.014311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014306 2579 scope.go:117] "RemoveContainer" containerID="2ee0b3792948da4a27de9828d86e037fc00f1570ea2ff7f970f946dc6ad55ce7" Sep 16 14:08:37.014645 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.014624 2579 scope.go:117] "RemoveContainer" containerID="55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76" Sep 16 14:08:37.015050 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:37.014841 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llzzn_openshift-console-operator(d9f92411-315d-4c2e-ab0c-22332bf4c8c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" podUID="d9f92411-315d-4c2e-ab0c-22332bf4c8c7" Sep 16 14:08:37.018204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.018180 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5" containerID="1e1fdc67604f1292f4ae4ac1a31c361bbb144a17ad6ee25210ae18e0f14aa1b7" exitCode=0 Sep 16 14:08:37.018319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.018294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerDied","Data":"1e1fdc67604f1292f4ae4ac1a31c361bbb144a17ad6ee25210ae18e0f14aa1b7"} Sep 16 14:08:37.142912 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.142865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.247163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.247130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.343635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.343532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.443866 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.443833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:37.444084 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.443896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:37.444084 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:37.443977 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 14:08:37.444084 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:37.444039 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:38.444024629 +0000 UTC m=+49.021385763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : configmap "kube-rbac-proxy" not found Sep 16 14:08:37.444084 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:37.444070 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 14:08:37.444292 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:37.444120 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:38.444104442 +0000 UTC m=+49.021465578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : secret "insights-runtime-extractor-tls" not found Sep 16 14:08:37.444486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.444454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.544751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.544718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.645177 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.645086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.758149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.758117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.866995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.866964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:37.983398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:37.983368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:38.028015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.027974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" event={"ID":"0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5","Type":"ContainerStarted","Data":"d0652d921b06aefaa71e50643620598cc1bcdafbf0b69122410195d27eaebff0"} Sep 16 14:08:38.029490 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.029471 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:08:38.029797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.029780 2579 scope.go:117] "RemoveContainer" containerID="55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76" Sep 16 14:08:38.029936 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:38.029919 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llzzn_openshift-console-operator(d9f92411-315d-4c2e-ab0c-22332bf4c8c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" podUID="d9f92411-315d-4c2e-ab0c-22332bf4c8c7" Sep 16 14:08:38.053055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.053010 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xh5lp" podStartSLOduration=7.517009552 podStartE2EDuration="47.052991328s" totalImagesPullingTime="39.535981776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:38.051590309 +0000 UTC m=+48.628951485" watchObservedRunningTime="2026-09-16 14:08:38.052991328 +0000 UTC m=+48.630352480" Sep 16 14:08:38.162282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.162242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:38.264007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.263913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:38.329449 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.329419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:38.452186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.452146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:38.452381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.452207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:38.452381 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:38.452281 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 14:08:38.452381 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:38.452337 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 14:08:38.452381 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:38.452351 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.452334867 +0000 UTC m=+51.029695996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : configmap "kube-rbac-proxy" not found Sep 16 14:08:38.452539 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:38.452387 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:40.452372101 +0000 UTC m=+51.029733231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : secret "insights-runtime-extractor-tls" not found Sep 16 14:08:38.509522 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:38.509494 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4rqpt_87d538d9-4506-4269-8ec6-5d4d1c8e402a/dns-node-resolver/0.log" Sep 16 14:08:39.267593 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:39.267556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:39.360203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:39.360164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:39.510017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:39.509987 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b97t4_54f5582a-0f28-4fbd-a29b-cc8b94f968b1/node-ca/0.log" Sep 16 14:08:40.311377 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.311346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-66tjl_2cce6995-91cf-4a57-87ca-ee871b0751b8/serve-healthcheck-canary/0.log" Sep 16 14:08:40.469384 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.469353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:40.469546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.469396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:40.469546 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.469477 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 14:08:40.469546 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.469544 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:44.469526648 +0000 UTC m=+55.046887798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : configmap "kube-rbac-proxy" not found Sep 16 14:08:40.469741 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.469554 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 14:08:40.469741 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.469587 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:44.469576554 +0000 UTC m=+55.046937686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : secret "insights-runtime-extractor-tls" not found Sep 16 14:08:40.584822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.584729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:40.671896 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.671865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:40.672020 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.672007 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 14:08:40.672081 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.672060 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls podName:cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.672046992 +0000 UTC m=+67.249408122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z8qxd" (UID: "cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c") : secret "samples-operator-tls" not found Sep 16 14:08:40.773110 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.773080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:40.773243 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.773125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:40.773243 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.773182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773243 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773262 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc95d58db-9c64m: secret "image-registry-tls" not found Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773289 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773300 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.773287351 +0000 UTC m=+67.350648481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : configmap references non-existent config key: service-ca.crt Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773317 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls podName:6f786f66-bf43-47a2-a5a5-c162fe54b013 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.773311019 +0000 UTC m=+67.350672149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls") pod "image-registry-dc95d58db-9c64m" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013") : secret "image-registry-tls" not found Sep 16 14:08:40.773343 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.773333 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs podName:4e8061e8-6358-4d13-bc50-c5d667cd0d4a nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.773321786 +0000 UTC m=+67.350682916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs") pod "router-default-685877fd7d-tp56g" (UID: "4e8061e8-6358-4d13-bc50-c5d667cd0d4a") : secret "router-metrics-certs-default" not found Sep 16 14:08:40.873956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.873865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:40.873956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:40.873927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:40.874141 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.874007 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:40.874141 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.874054 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 14:08:40.874141 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.874069 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls podName:574a4a71-86e6-4ed6-9080-404c193e5884 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.874055514 +0000 UTC m=+67.451416647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fcdzq" (UID: "574a4a71-86e6-4ed6-9080-404c193e5884") : secret "cluster-monitoring-operator-tls" not found Sep 16 14:08:40.874141 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:40.874091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls podName:9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1 nodeName:}" failed. No retries permitted until 2026-09-16 14:08:56.874079513 +0000 UTC m=+67.451440643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls") pod "dns-default-qmqk4" (UID: "9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1") : secret "dns-default-metrics-tls" not found Sep 16 14:08:41.599896 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:41.599858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:43.442283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:43.442244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:44.503725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.503691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:44.504113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.503737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:44.504113 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:44.503829 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 14:08:44.504113 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:44.503884 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls podName:2d15e0cc-390c-4f9f-9671-02b6ca82cd9c nodeName:}" failed. No retries permitted until 2026-09-16 14:08:52.503870825 +0000 UTC m=+63.081231955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4sqqr" (UID: "2d15e0cc-390c-4f9f-9671-02b6ca82cd9c") : secret "insights-runtime-extractor-tls" not found Sep 16 14:08:44.504318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.504293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:44.697548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.697513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:44.798553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.798468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:44.904754 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:44.904724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:45.037773 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:45.037734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:45.281228 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:45.281177 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:45.281385 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:45.281255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:45.281624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:45.281594 2579 scope.go:117] "RemoveContainer" containerID="55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76" Sep 16 14:08:45.281796 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:08:45.281776 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llzzn_openshift-console-operator(d9f92411-315d-4c2e-ab0c-22332bf4c8c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" podUID="d9f92411-315d-4c2e-ab0c-22332bf4c8c7" Sep 16 14:08:45.788736 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:45.788696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:46.385319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:46.385285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:46.611875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:46.611839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:46.707780 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:46.707751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.475943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:48.475901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.541526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:48.541494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.609596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:48.609564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.799133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:48.799047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.987627 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:48.987573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:49.050492 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:49.050418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:49.587749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:49.587718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:50.015134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:50.015101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:50.813723 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:50.813692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.092001 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:51.091916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.284235 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:51.284196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.415392 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:51.415305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.637425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:51.637383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.939866 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:51.939840 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27422" Sep 16 14:08:52.001497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.001460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.262134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.262049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.406014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.405982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.505790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.505744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.573834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.573760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:52.576286 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.576262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d15e0cc-390c-4f9f-9671-02b6ca82cd9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4sqqr\" (UID: \"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c\") " pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:52.699909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.699878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c7jvw\"" Sep 16 14:08:52.707338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.707316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4sqqr" Sep 16 14:08:52.792812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.792782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.978984 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.978960 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4sqqr"] Sep 16 14:08:52.981458 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:52.981431 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d15e0cc_390c_4f9f_9671_02b6ca82cd9c.slice/crio-6a6c80b824f1485888f60fb0327d08a63c890d1fb9ab17d9a46d9a7bc2dc0f2e WatchSource:0}: Error finding container 6a6c80b824f1485888f60fb0327d08a63c890d1fb9ab17d9a46d9a7bc2dc0f2e: Status 404 returned error can't find the container with id 6a6c80b824f1485888f60fb0327d08a63c890d1fb9ab17d9a46d9a7bc2dc0f2e Sep 16 14:08:52.998769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:52.998736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:53.075143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:53.075109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4sqqr" event={"ID":"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c","Type":"ContainerStarted","Data":"113b4ea36eb585b96b35966162c1c8ce2073d080c9b9a84d064ba21ca7089a62"} Sep 16 14:08:53.075313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:53.075169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4sqqr" event={"ID":"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c","Type":"ContainerStarted","Data":"6a6c80b824f1485888f60fb0327d08a63c890d1fb9ab17d9a46d9a7bc2dc0f2e"} Sep 16 14:08:53.198376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:53.198344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:53.593178 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:53.593148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:53.993496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:53.993457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:54.080527 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:54.080494 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4sqqr" event={"ID":"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c","Type":"ContainerStarted","Data":"c9973c90e4b09bfa705a05891929d0b0323e9050d2c5d145fc82615db2e66086"} Sep 16 14:08:54.200390 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:54.200361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:55.000016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:54.999985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:55.202417 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.202378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:55.701330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.701299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:55.701440 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.701432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:55.704097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.704081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 14:08:55.705087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.705069 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 14:08:55.714196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.714177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb787601-e0a7-43ca-a6c9-6f66b529995b-original-pull-secret\") pod \"global-pull-secret-syncer-rbr5h\" (UID: \"bb787601-e0a7-43ca-a6c9-6f66b529995b\") " pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:55.714324 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.714309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1-metrics-certs\") pod \"network-metrics-daemon-7lx7g\" (UID: \"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1\") " pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:55.802649 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.802563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:55.805074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.805046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvf2n\" (UniqueName: \"kubernetes.io/projected/573364cf-e95e-4380-9e10-8c0265047660-kube-api-access-hvf2n\") pod \"network-check-target-5cgs2\" (UID: \"573364cf-e95e-4380-9e10-8c0265047660\") " pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:55.825165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.825144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbr5h" Sep 16 14:08:55.836725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.836703 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x4gh8\"" Sep 16 14:08:55.843905 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.843882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx7g" Sep 16 14:08:55.847381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.847366 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8mpwg\"" Sep 16 14:08:55.850206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:55.850186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:56.030510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.030473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:56.088406 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.088341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4sqqr" event={"ID":"2d15e0cc-390c-4f9f-9671-02b6ca82cd9c","Type":"ContainerStarted","Data":"802f8e2b367f00fcc0e49325aadf86c4b48ff46aac2a075897b2bd1f658f3a0f"} Sep 16 14:08:56.101896 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.101873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbr5h"] Sep 16 14:08:56.105560 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:56.105536 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb787601_e0a7_43ca_a6c9_6f66b529995b.slice/crio-745090c9a8213d0647dd78989f74e57778eeb4265a4f4e711ef6d009bb9a67bc WatchSource:0}: Error finding container 745090c9a8213d0647dd78989f74e57778eeb4265a4f4e711ef6d009bb9a67bc: Status 404 returned error can't find the container with id 745090c9a8213d0647dd78989f74e57778eeb4265a4f4e711ef6d009bb9a67bc Sep 16 14:08:56.114548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.112976 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4sqqr" podStartSLOduration=17.667048044 podStartE2EDuration="20.112956767s" totalImagesPullingTime="2.445908723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:08:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:56.112153799 +0000 UTC m=+66.689514968" watchObservedRunningTime="2026-09-16 14:08:56.112956767 +0000 UTC m=+66.690317921" Sep 16 14:08:56.130388 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.130364 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lx7g"] Sep 16 14:08:56.133286 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:56.133264 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fffcd6_2eeb_4baa_b2f5_c4d6d832a4b1.slice/crio-ddc0e59234ed2dfbf144da7725b7d7579552a530c8c90d671df937d0db05fcea WatchSource:0}: Error finding container ddc0e59234ed2dfbf144da7725b7d7579552a530c8c90d671df937d0db05fcea: Status 404 returned error can't find the container with id ddc0e59234ed2dfbf144da7725b7d7579552a530c8c90d671df937d0db05fcea Sep 16 14:08:56.165049 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.165025 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cgs2"] Sep 16 14:08:56.167557 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:56.167537 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573364cf_e95e_4380_9e10_8c0265047660.slice/crio-f27fb84565058d3896cca8871a3aa4fbb311b1df780ca492fe885a99ce8bd10c WatchSource:0}: Error finding container f27fb84565058d3896cca8871a3aa4fbb311b1df780ca492fe885a99ce8bd10c: Status 404 returned error can't find the container with id f27fb84565058d3896cca8871a3aa4fbb311b1df780ca492fe885a99ce8bd10c Sep 16 14:08:56.713684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.713653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:56.716088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.716061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z8qxd\" (UID: \"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:56.811720 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.811691 2579 scope.go:117] "RemoveContainer" containerID="55d3c80405c5c491b37490bf76de096a163d4c1a453e702a0a3db30f6919af76" Sep 16 14:08:56.815747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.814289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:56.815747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.814364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:56.815747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.814396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:56.815747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.815420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-service-ca-bundle\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:56.819026 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.819003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"image-registry-dc95d58db-9c64m\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:56.819571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.819548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8061e8-6358-4d13-bc50-c5d667cd0d4a-metrics-certs\") pod \"router-default-685877fd7d-tp56g\" (UID: \"4e8061e8-6358-4d13-bc50-c5d667cd0d4a\") " pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:56.915027 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.914997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:56.915195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.915085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:56.920164 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.920119 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1-metrics-tls\") pod \"dns-default-qmqk4\" (UID: \"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1\") " pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:56.920793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.920772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/574a4a71-86e6-4ed6-9080-404c193e5884-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fcdzq\" (UID: \"574a4a71-86e6-4ed6-9080-404c193e5884\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:56.947651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.947627 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-96brr\"" Sep 16 14:08:56.955109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.954899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" Sep 16 14:08:56.973895 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.973703 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ffjnt\"" Sep 16 14:08:56.980768 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:56.980331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:57.020980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.016253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-82498\"" Sep 16 14:08:57.021636 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.021304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:57.066705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.066481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nrxcf\"" Sep 16 14:08:57.074633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.074566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" Sep 16 14:08:57.096071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.096037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbr5h" event={"ID":"bb787601-e0a7-43ca-a6c9-6f66b529995b","Type":"ContainerStarted","Data":"745090c9a8213d0647dd78989f74e57778eeb4265a4f4e711ef6d009bb9a67bc"} Sep 16 14:08:57.100453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.098692 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:08:57.100453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.098791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" event={"ID":"d9f92411-315d-4c2e-ab0c-22332bf4c8c7","Type":"ContainerStarted","Data":"5b94313a4c77c43f9a9d1476f52eccd25a109fcbdbda40bd1168477ed8d21559"} Sep 16 14:08:57.100453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.099740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:57.102035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.101961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cgs2" event={"ID":"573364cf-e95e-4380-9e10-8c0265047660","Type":"ContainerStarted","Data":"a1a3fde5839c3d89ab9552bf81c16e5f01e192a53a16e8f01a9afc17681ba724"} Sep 16 14:08:57.102035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.101998 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cgs2" event={"ID":"573364cf-e95e-4380-9e10-8c0265047660","Type":"ContainerStarted","Data":"f27fb84565058d3896cca8871a3aa4fbb311b1df780ca492fe885a99ce8bd10c"} Sep 16 14:08:57.102035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.102022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:08:57.104096 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.104076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx7g" event={"ID":"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1","Type":"ContainerStarted","Data":"ddc0e59234ed2dfbf144da7725b7d7579552a530c8c90d671df937d0db05fcea"} Sep 16 14:08:57.155070 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.155018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" podStartSLOduration=34.267418168 podStartE2EDuration="44.155003187s" totalImagesPullingTime="9.887585019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:57.124312866 +0000 UTC m=+67.701674019" watchObservedRunningTime="2026-09-16 14:08:57.155003187 +0000 UTC m=+67.732364340" Sep 16 14:08:57.156044 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.156012 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5cgs2" podStartSLOduration=67.156000573 podStartE2EDuration="1m7.156000573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:57.148927541 +0000 UTC m=+67.726288694" watchObservedRunningTime="2026-09-16 14:08:57.156000573 +0000 UTC m=+67.733361726" Sep 16 14:08:57.157431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.157411 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8lck4\"" Sep 16 14:08:57.158481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.158452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmqk4" Sep 16 14:08:57.251481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.251453 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd"] Sep 16 14:08:57.277480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.277450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:08:57.298241 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:57.298208 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f786f66_bf43_47a2_a5a5_c162fe54b013.slice/crio-f97ccacef2e32a94120bdebd2cfb32eada2924ba2ea2987679b6dc5c55f8a632 WatchSource:0}: Error finding container f97ccacef2e32a94120bdebd2cfb32eada2924ba2ea2987679b6dc5c55f8a632: Status 404 returned error can't find the container with id f97ccacef2e32a94120bdebd2cfb32eada2924ba2ea2987679b6dc5c55f8a632 Sep 16 14:08:57.555847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.555816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-llzzn" Sep 16 14:08:57.626787 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.626756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-685877fd7d-tp56g"] Sep 16 14:08:57.634649 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:08:57.634594 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8061e8_6358_4d13_bc50_c5d667cd0d4a.slice/crio-02fe3ef0fa3c6a9baa054b231a1cc9c22a0c5e00bdf49186a4d9b842b39a061c WatchSource:0}: Error finding container 02fe3ef0fa3c6a9baa054b231a1cc9c22a0c5e00bdf49186a4d9b842b39a061c: Status 404 returned error can't find the container with id 02fe3ef0fa3c6a9baa054b231a1cc9c22a0c5e00bdf49186a4d9b842b39a061c Sep 16 14:08:57.645308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.645198 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmqk4"] Sep 16 14:08:57.674501 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.674053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq"] Sep 16 14:08:57.826064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.825972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:57.933563 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:57.933530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:58.075282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.075246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:58.110862 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.110502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx7g" event={"ID":"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1","Type":"ContainerStarted","Data":"54fa4ecb236b6d563154f0f983feb8b4bb898ff2eb675ec883c9d23b624c9fc4"} Sep 16 14:08:58.110862 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.110565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx7g" event={"ID":"01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1","Type":"ContainerStarted","Data":"4734352df5d647b1b11620f7c251a7cda2789934b9df61352c25a4912f917882"} Sep 16 14:08:58.112002 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.111974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" event={"ID":"574a4a71-86e6-4ed6-9080-404c193e5884","Type":"ContainerStarted","Data":"25559efe63fdc9ecb34dae9aa5df3b0252facbae0edc62ff0e4782ae3f03732f"} Sep 16 14:08:58.113398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.113372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmqk4" event={"ID":"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1","Type":"ContainerStarted","Data":"256897d3752041c43ce66a80a1060f48b9c8f2aaff60b133c25a4ef491b91583"} Sep 16 14:08:58.114685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.114644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" event={"ID":"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c","Type":"ContainerStarted","Data":"6c39495a00ab1997b2cf45c8a1cebdfcda887304645f7bd1cfa5865e18472126"} Sep 16 14:08:58.116343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.116310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-685877fd7d-tp56g" event={"ID":"4e8061e8-6358-4d13-bc50-c5d667cd0d4a","Type":"ContainerStarted","Data":"fd13fa27aa91d8f485ae5cfd67b104ed799cf07c2535f33c94485b4caba7f4f4"} Sep 16 14:08:58.116448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.116357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-685877fd7d-tp56g" event={"ID":"4e8061e8-6358-4d13-bc50-c5d667cd0d4a","Type":"ContainerStarted","Data":"02fe3ef0fa3c6a9baa054b231a1cc9c22a0c5e00bdf49186a4d9b842b39a061c"} Sep 16 14:08:58.118068 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.118042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" event={"ID":"6f786f66-bf43-47a2-a5a5-c162fe54b013","Type":"ContainerStarted","Data":"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964"} Sep 16 14:08:58.118187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.118090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" event={"ID":"6f786f66-bf43-47a2-a5a5-c162fe54b013","Type":"ContainerStarted","Data":"f97ccacef2e32a94120bdebd2cfb32eada2924ba2ea2987679b6dc5c55f8a632"} Sep 16 14:08:58.143567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.143522 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7lx7g" podStartSLOduration=66.912472905 podStartE2EDuration="1m8.143509846s" totalImagesPullingTime="1.231036941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:58.14252511 +0000 UTC m=+68.719886262" watchObservedRunningTime="2026-09-16 14:08:58.143509846 +0000 UTC m=+68.720870997" Sep 16 14:08:58.169957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.169911 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" podStartSLOduration=67.169897438 podStartE2EDuration="1m7.169897438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:58.168707778 +0000 UTC m=+68.746068929" watchObservedRunningTime="2026-09-16 14:08:58.169897438 +0000 UTC m=+68.747258591" Sep 16 14:08:58.210776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.210731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-685877fd7d-tp56g" podStartSLOduration=45.210716993 podStartE2EDuration="45.210716993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:58.207759183 +0000 UTC m=+68.785120335" watchObservedRunningTime="2026-09-16 14:08:58.210716993 +0000 UTC m=+68.788078144" Sep 16 14:08:58.232531 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.232507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:58.334040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.334003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:58.443090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.443062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:58.827114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:58.827032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:59.022861 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:59.022828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:59.025790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:59.025564 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:59.125319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:59.125243 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:08:59.125792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:59.125289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:08:59.126744 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:08:59.126488 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-685877fd7d-tp56g" Sep 16 14:09:00.392422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:00.392386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.496499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:00.496463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.593587 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:00.593549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.927777 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:00.927736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.994787 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:00.994749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:02.301378 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:02.301321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:02.362410 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:02.362319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:03.613082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:03.612991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:03.714227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:03.714197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:03.797758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:03.797725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:04.082555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.082518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:04.149139 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.149095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" event={"ID":"574a4a71-86e6-4ed6-9080-404c193e5884","Type":"ContainerStarted","Data":"58bb77e607174bed99497c6851558bef22c0a9d3786f7a5176abb4bd9c850e24"} Sep 16 14:09:04.150910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.150881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmqk4" event={"ID":"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1","Type":"ContainerStarted","Data":"130872f1088944f6f996ed4ceca9049cb7730e69843f441fd84c73cf0152446e"} Sep 16 14:09:04.151054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.150935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qmqk4" Sep 16 14:09:04.151054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.150960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmqk4" event={"ID":"9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1","Type":"ContainerStarted","Data":"8dbe85ae5a930ebdb9d3bb90e844a7bc21c1725513109e5a92c7664b50c5e5c4"} Sep 16 14:09:04.152695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.152670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" event={"ID":"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c","Type":"ContainerStarted","Data":"8fb603e598b04796dcdc17963b2ed85b7dc41febe8fa084606206d8e57312442"} Sep 16 14:09:04.152818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.152722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" event={"ID":"cfa0bdbe-1fd8-4cd1-9d72-db4dc56b1a6c","Type":"ContainerStarted","Data":"d26044adbe1d26b2d45ebd2a8fc77faf6cfdd92d3a1491822b67eb707dc74ce4"} Sep 16 14:09:04.154145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.154122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbr5h" event={"ID":"bb787601-e0a7-43ca-a6c9-6f66b529995b","Type":"ContainerStarted","Data":"71d00915655ed37aaf940e141d4220871b65e517c47c8a84d91deee1b8907fe0"} Sep 16 14:09:04.167211 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.167174 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fcdzq" podStartSLOduration=45.761455418 podStartE2EDuration="51.167161787s" totalImagesPullingTime="5.405706369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:04.165664013 +0000 UTC m=+74.743025166" watchObservedRunningTime="2026-09-16 14:09:04.167161787 +0000 UTC m=+74.744522940" Sep 16 14:09:04.183288 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.183248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z8qxd" podStartSLOduration=45.472896651 podStartE2EDuration="51.183235689s" totalImagesPullingTime="5.710339038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:04.182730491 +0000 UTC m=+74.760091622" watchObservedRunningTime="2026-09-16 14:09:04.183235689 +0000 UTC m=+74.760596841" Sep 16 14:09:04.196826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.196781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rbr5h" podStartSLOduration=66.233660642 podStartE2EDuration="1m13.196768827s" totalImagesPullingTime="6.963108185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:04.196081695 +0000 UTC m=+74.773442847" watchObservedRunningTime="2026-09-16 14:09:04.196768827 +0000 UTC m=+74.774129972" Sep 16 14:09:04.213462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.213429 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qmqk4" podStartSLOduration=34.797707695 podStartE2EDuration="40.213417008s" totalImagesPullingTime="5.415709313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:04.212847654 +0000 UTC m=+74.790208806" watchObservedRunningTime="2026-09-16 14:09:04.213417008 +0000 UTC m=+74.790778159" Sep 16 14:09:04.365542 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.365459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:04.428621 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.428579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:04.498319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.498282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:04.758833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.758800 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:04.779687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.779661 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:04.779826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.779748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.782550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.782528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 14:09:04.782716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.782680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 14:09:04.782788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.782773 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 14:09:04.784000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.783840 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 14:09:04.784000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.783860 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 14:09:04.784000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.783884 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 14:09:04.784000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.783964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 14:09:04.784000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.783976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w628n\"" Sep 16 14:09:04.888086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.888086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxfp\" (UniqueName: \"kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.888278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.888278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.888278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.888278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.888210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989494 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989494 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxfp\" (UniqueName: \"kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.989770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.989741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.990258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.990228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.990363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.990289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.990363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.990344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.992242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.992224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.992334 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.992231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:04.998012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:04.997979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxfp\" (UniqueName: \"kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp\") pod \"console-69d6c6bd59-8fk4j\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:05.089986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:05.089895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:05.390112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:05.390085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:05.392786 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:09:05.392758 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded90a5e6_9d39_4aff_96d0_826a563f64fc.slice/crio-ac8cd7ccbe98e61476afecf816d49a99937aea08c24733b036ed331bdb1186f3 WatchSource:0}: Error finding container ac8cd7ccbe98e61476afecf816d49a99937aea08c24733b036ed331bdb1186f3: Status 404 returned error can't find the container with id ac8cd7ccbe98e61476afecf816d49a99937aea08c24733b036ed331bdb1186f3 Sep 16 14:09:06.163922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.163881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d6c6bd59-8fk4j" event={"ID":"ed90a5e6-9d39-4aff-96d0-826a563f64fc","Type":"ContainerStarted","Data":"ac8cd7ccbe98e61476afecf816d49a99937aea08c24733b036ed331bdb1186f3"} Sep 16 14:09:06.619162 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.619121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:06.709926 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.709886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:06.810435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.810374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:06.863401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.863366 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wpj97"] Sep 16 14:09:06.878709 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.878589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:06.880668 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.880643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wpj97"] Sep 16 14:09:06.881556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.881533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 14:09:06.881814 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.881796 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 14:09:06.882659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.882643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5rfhf\"" Sep 16 14:09:06.882792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.882649 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 14:09:06.901600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:06.901577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.006526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.006500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.006684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.006542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tqn\" (UniqueName: \"kubernetes.io/projected/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-kube-api-access-x6tqn\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.006684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.006563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.006684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.006595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.107661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.107625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tqn\" (UniqueName: \"kubernetes.io/projected/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-kube-api-access-x6tqn\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.107843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.107672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.107843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.107711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.107843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.107765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.107964 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:07.107855 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 14:09:07.107964 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:07.107914 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls podName:f1d2ab1c-6396-484a-a3d0-383d00ef4b5d nodeName:}" failed. No retries permitted until 2026-09-16 14:09:07.607895474 +0000 UTC m=+78.185256603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-wpj97" (UID: "f1d2ab1c-6396-484a-a3d0-383d00ef4b5d") : secret "prometheus-operator-tls" not found Sep 16 14:09:07.108370 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.108342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.110575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.110549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.119893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.119871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tqn\" (UniqueName: \"kubernetes.io/projected/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-kube-api-access-x6tqn\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.230157 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.230120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.332532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.332499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.612623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.612512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.615402 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.615374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1d2ab1c-6396-484a-a3d0-383d00ef4b5d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wpj97\" (UID: \"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:07.791515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:07.791473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" Sep 16 14:09:08.777889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:08.777816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wpj97"] Sep 16 14:09:08.845581 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:09:08.845536 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d2ab1c_6396_484a_a3d0_383d00ef4b5d.slice/crio-b4ee4e3155c81a918033b771584c3b4078df18cb8ae7e9ad00b4d2066aab1b96 WatchSource:0}: Error finding container b4ee4e3155c81a918033b771584c3b4078df18cb8ae7e9ad00b4d2066aab1b96: Status 404 returned error can't find the container with id b4ee4e3155c81a918033b771584c3b4078df18cb8ae7e9ad00b4d2066aab1b96 Sep 16 14:09:08.940387 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:08.940098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:09.033588 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.033499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:09.177488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.177450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d6c6bd59-8fk4j" event={"ID":"ed90a5e6-9d39-4aff-96d0-826a563f64fc","Type":"ContainerStarted","Data":"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e"} Sep 16 14:09:09.178685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.178666 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" event={"ID":"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d","Type":"ContainerStarted","Data":"b4ee4e3155c81a918033b771584c3b4078df18cb8ae7e9ad00b4d2066aab1b96"} Sep 16 14:09:09.197901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.197854 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69d6c6bd59-8fk4j" podStartSLOduration=1.716617096 podStartE2EDuration="5.197838253s" totalImagesPullingTime="3.481221157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:09.19616916 +0000 UTC m=+79.773530311" watchObservedRunningTime="2026-09-16 14:09:09.197838253 +0000 UTC m=+79.775199405" Sep 16 14:09:09.456720 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.456672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:09.556982 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.556943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:09.658179 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:09.658143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:10.748512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:10.748439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:10.895889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:10.895856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:11.127843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:11.127761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:11.191856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:11.191819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" event={"ID":"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d","Type":"ContainerStarted","Data":"e5a408e7b9b0ae28a7e949ec94aab136d2abbdab6451e5cf3a10d6bbdb2e3363"} Sep 16 14:09:11.192018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:11.191915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" event={"ID":"f1d2ab1c-6396-484a-a3d0-383d00ef4b5d","Type":"ContainerStarted","Data":"f5a119baec087e9940cbc962b3bac945c9ac10dc9d2da0cc08c0c61502fc4121"} Sep 16 14:09:11.218486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:11.218434 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wpj97" podStartSLOduration=3.632990767 podStartE2EDuration="5.21841817s" totalImagesPullingTime="1.585427403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:11.2167387 +0000 UTC m=+81.794099852" watchObservedRunningTime="2026-09-16 14:09:11.21841817 +0000 UTC m=+81.795779346" Sep 16 14:09:12.893219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:12.893179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:12.996022 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:12.995985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.098856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.098806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.203712 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.203672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.253570 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.253543 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4nbm"] Sep 16 14:09:13.258091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.258071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.262677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.262646 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g65fn\"" Sep 16 14:09:13.262677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.262673 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 14:09:13.262821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.262676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 14:09:13.263976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.263960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 14:09:13.318094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.318069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.355582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-root\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355711 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-metrics-client-ca\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355711 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-wtmp\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355711 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-sys\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-textfile\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.355946 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.355852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjbf\" (UniqueName: \"kubernetes.io/projected/0729bf3f-a59a-474f-899d-ed06496f7884-kube-api-access-zkjbf\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.456940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.456973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-textfile\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjbf\" (UniqueName: \"kubernetes.io/projected/0729bf3f-a59a-474f-899d-ed06496f7884-kube-api-access-zkjbf\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-root\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-metrics-client-ca\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-wtmp\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-sys\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-root\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-sys\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457550 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:13.457295 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 14:09:13.457550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-wtmp\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-textfile\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457550 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:13.457351 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls podName:0729bf3f-a59a-474f-899d-ed06496f7884 nodeName:}" failed. No retries permitted until 2026-09-16 14:09:13.957332913 +0000 UTC m=+84.534694043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls") pod "node-exporter-c4nbm" (UID: "0729bf3f-a59a-474f-899d-ed06496f7884") : secret "node-exporter-tls" not found Sep 16 14:09:13.457724 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.457724 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.457649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0729bf3f-a59a-474f-899d-ed06496f7884-metrics-client-ca\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.459593 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.459574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.465429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.465407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjbf\" (UniqueName: \"kubernetes.io/projected/0729bf3f-a59a-474f-899d-ed06496f7884-kube-api-access-zkjbf\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.587319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.587286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.684581 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.684548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.786102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.786010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.885323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.885291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:13.960620 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.960572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.963040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.963016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0729bf3f-a59a-474f-899d-ed06496f7884-node-exporter-tls\") pod \"node-exporter-c4nbm\" (UID: \"0729bf3f-a59a-474f-899d-ed06496f7884\") " pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:13.997573 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:13.997542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:14.161531 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.161443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qmqk4" Sep 16 14:09:14.166520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.166493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4nbm" Sep 16 14:09:14.188458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.188428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:14.204501 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.204477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4nbm" event={"ID":"0729bf3f-a59a-474f-899d-ed06496f7884","Type":"ContainerStarted","Data":"79e5f6b1053711d491b4dcd1e91825280ee45d1943378742fff710e0b70a705a"} Sep 16 14:09:14.277137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.277100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:14.382502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.382457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:14.477568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:14.477527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:15.090180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.090148 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:15.090665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.090421 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:15.096394 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.096369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:15.213115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.213085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:15.462392 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.462336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:15.570940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.570906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:15.671308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.671266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:15.771792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:15.771712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:16.165344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.165250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:16.212999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.212971 2579 generic.go:356] "Generic (PLEG): container finished" podID="0729bf3f-a59a-474f-899d-ed06496f7884" containerID="50d902d976a088797d5a00de7eb40c584e6cb0db2c235df5730d4bfa36a8a8df" exitCode=0 Sep 16 14:09:16.213154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.213065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4nbm" event={"ID":"0729bf3f-a59a-474f-899d-ed06496f7884","Type":"ContainerDied","Data":"50d902d976a088797d5a00de7eb40c584e6cb0db2c235df5730d4bfa36a8a8df"} Sep 16 14:09:16.236955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.236910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:16.298533 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.298508 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-769866f4f6-sh6km"] Sep 16 14:09:16.302923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.302904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.305642 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 14:09:16.305746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pvpvt\"" Sep 16 14:09:16.305810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305747 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 14:09:16.305810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305784 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f0a2j00muinon\"" Sep 16 14:09:16.305810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 14:09:16.305955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305907 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 14:09:16.306005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.305981 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 14:09:16.314081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.314059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-769866f4f6-sh6km"] Sep 16 14:09:16.478589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzs9r\" (UniqueName: \"kubernetes.io/projected/117d62b8-9643-479e-ba7d-c2ee06d97f04-kube-api-access-rzs9r\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117d62b8-9643-479e-ba7d-c2ee06d97f04-metrics-client-ca\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-grpc-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.478940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.478927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579492 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117d62b8-9643-479e-ba7d-c2ee06d97f04-metrics-client-ca\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-grpc-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzs9r\" (UniqueName: \"kubernetes.io/projected/117d62b8-9643-479e-ba7d-c2ee06d97f04-kube-api-access-rzs9r\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.579933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.579717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.580592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.580434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117d62b8-9643-479e-ba7d-c2ee06d97f04-metrics-client-ca\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.582371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.582337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.582371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.582364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.582944 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.582919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-grpc-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.582944 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.582938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-tls\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.583111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.583077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.583174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.583105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/117d62b8-9643-479e-ba7d-c2ee06d97f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.587978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.587954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzs9r\" (UniqueName: \"kubernetes.io/projected/117d62b8-9643-479e-ba7d-c2ee06d97f04-kube-api-access-rzs9r\") pod \"thanos-querier-769866f4f6-sh6km\" (UID: \"117d62b8-9643-479e-ba7d-c2ee06d97f04\") " pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.604407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.604383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:16.625274 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.625250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:16.705254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.705212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:16.904966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:16.904945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-769866f4f6-sh6km"] Sep 16 14:09:16.907393 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:09:16.907363 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117d62b8_9643_479e_ba7d_c2ee06d97f04.slice/crio-91cf7b6ab90477c6dec294b43802c966fb757090900aacacc0be2d5c426c28a6 WatchSource:0}: Error finding container 91cf7b6ab90477c6dec294b43802c966fb757090900aacacc0be2d5c426c28a6: Status 404 returned error can't find the container with id 91cf7b6ab90477c6dec294b43802c966fb757090900aacacc0be2d5c426c28a6 Sep 16 14:09:17.146564 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.146533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:17.219144 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.219110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4nbm" event={"ID":"0729bf3f-a59a-474f-899d-ed06496f7884","Type":"ContainerStarted","Data":"3ad1be6522a358ddd258586d893fbcde38b24717c9504364188a82b7ca11d1f5"} Sep 16 14:09:17.219559 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.219164 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4nbm" event={"ID":"0729bf3f-a59a-474f-899d-ed06496f7884","Type":"ContainerStarted","Data":"c01b99d84d271f5bcb332329851a8e0a7285fd57ecc586bb3e975e1bf5174a5a"} Sep 16 14:09:17.220457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.220435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"91cf7b6ab90477c6dec294b43802c966fb757090900aacacc0be2d5c426c28a6"} Sep 16 14:09:17.252730 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.252689 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4nbm" podStartSLOduration=2.827512607 podStartE2EDuration="4.25267322s" totalImagesPullingTime="1.353200549s" totalInitContainerRuntime="71.960064ms" isStatefulPod=true podCreationTimestamp="2026-09-16 14:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:17.251044561 +0000 UTC m=+87.828405711" watchObservedRunningTime="2026-09-16 14:09:17.25267322 +0000 UTC m=+87.830034411" Sep 16 14:09:17.352207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:17.352179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.030789 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.030751 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl"] Sep 16 14:09:18.034260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.034228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.037093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.037050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 14:09:18.037419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.037402 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mxspf\"" Sep 16 14:09:18.047714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.047691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl"] Sep 16 14:09:18.195093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.195053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d73451e-a0cb-4616-b497-844150383c23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.195256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.195113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24n2c\" (UniqueName: \"kubernetes.io/projected/1d73451e-a0cb-4616-b497-844150383c23-kube-api-access-24n2c\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.296214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.296137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d73451e-a0cb-4616-b497-844150383c23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.296214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.296192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24n2c\" (UniqueName: \"kubernetes.io/projected/1d73451e-a0cb-4616-b497-844150383c23-kube-api-access-24n2c\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.299264 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.299240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d73451e-a0cb-4616-b497-844150383c23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.306152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.306124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24n2c\" (UniqueName: \"kubernetes.io/projected/1d73451e-a0cb-4616-b497-844150383c23-kube-api-access-24n2c\") pod \"monitoring-plugin-5d85778b9d-x87gl\" (UID: \"1d73451e-a0cb-4616-b497-844150383c23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.347313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.347288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:18.627681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:18.626485 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl"] Sep 16 14:09:18.630983 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:09:18.630957 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d73451e_a0cb_4616_b497_844150383c23.slice/crio-8d7e7f94f5be9f3eeb7f64378fce215c5f29149c37d576695c0bbdc72bcd61c0 WatchSource:0}: Error finding container 8d7e7f94f5be9f3eeb7f64378fce215c5f29149c37d576695c0bbdc72bcd61c0: Status 404 returned error can't find the container with id 8d7e7f94f5be9f3eeb7f64378fce215c5f29149c37d576695c0bbdc72bcd61c0 Sep 16 14:09:19.194528 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:19.194491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:19.234703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:19.234653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" event={"ID":"1d73451e-a0cb-4616-b497-844150383c23","Type":"ContainerStarted","Data":"8d7e7f94f5be9f3eeb7f64378fce215c5f29149c37d576695c0bbdc72bcd61c0"} Sep 16 14:09:19.266412 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:19.266378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:19.500389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:19.500328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:19.614412 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:19.614226 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:20.134370 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.134342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:09:20.242234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.242199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"16d890b64ec6c4c7b28198de9b9b92a39aa51dbbc0ef26a8628b238b04047022"} Sep 16 14:09:20.242417 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.242264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"9676afc27216739998b93eae30bf02d138576f7d667b4595898bc4566589534d"} Sep 16 14:09:20.242417 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.242294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"a0b7d38cc524e6b4d4ebb59250fcd08167733eaff5056a351d19ac23e9f17ac5"} Sep 16 14:09:20.326790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.326752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:20.429715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:20.429686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:21.247463 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.247427 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" event={"ID":"1d73451e-a0cb-4616-b497-844150383c23","Type":"ContainerStarted","Data":"ed646e64037aea6fa0523c11e9f02f971d52293e0b5457444e4b75ef77197b86"} Sep 16 14:09:21.247954 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.247629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:21.252480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.252455 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" Sep 16 14:09:21.268366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.268317 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-x87gl" podStartSLOduration=1.446113597 podStartE2EDuration="3.268302842s" totalImagesPullingTime="1.822189245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:21.26714083 +0000 UTC m=+91.844501983" watchObservedRunningTime="2026-09-16 14:09:21.268302842 +0000 UTC m=+91.845663993" Sep 16 14:09:21.568268 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.568170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:21.643994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:21.643960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:22.279953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:22.279903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:22.354172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:22.354140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:09:22.380836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:22.380791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:22.486801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:22.486767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:22.585558 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:22.585508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:23.259194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.259161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"0ea30abd07793073d8cf45333c688ed704c15a756c0c2c19ce027996fcc06157"} Sep 16 14:09:23.259374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.259229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"d17c8c40ce835a1f73d4d349140be5b1b83252b75edf341b7299e58a52544864"} Sep 16 14:09:23.259374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.259263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" event={"ID":"117d62b8-9643-479e-ba7d-c2ee06d97f04","Type":"ContainerStarted","Data":"f03b2fda88f7d6547edac23cbcf56b67559fa699df5f3bb4dfeaaf4f51cf6468"} Sep 16 14:09:23.259374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.259341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:23.289637 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.289571 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" podStartSLOduration=1.998927347 podStartE2EDuration="7.289553886s" totalImagesPullingTime="5.290626539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:23.287882886 +0000 UTC m=+93.865244037" watchObservedRunningTime="2026-09-16 14:09:23.289553886 +0000 UTC m=+93.866915037" Sep 16 14:09:23.915482 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.915429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:23.921234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.921209 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:09:23.924440 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.924425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.934847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.934826 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 14:09:23.944620 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njmn\" (UniqueName: \"kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.944906 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.944831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:23.949915 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:23.949888 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:09:24.009974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.009931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:24.045684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.045850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.045850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.045850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.045850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njmn\" (UniqueName: \"kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.045850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.046112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.045895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.046466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.046440 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.046756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.046477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.046756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.046596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.046923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.046899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.048521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.048494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.048597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.048522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.055069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.055045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njmn\" (UniqueName: \"kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn\") pod \"console-9f5cb57f5-kldn8\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.233010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.232970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:24.269397 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.269372 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-769866f4f6-sh6km" Sep 16 14:09:24.531950 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:24.531869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:09:24.537959 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:09:24.537930 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d56b9b_0403_44c3_a2a8_2d3a9250c9ec.slice/crio-f325ddb885b0f5b8f201dbdaa6a873629c5cc28bffa9ab7d6039f3b73e9863c2 WatchSource:0}: Error finding container f325ddb885b0f5b8f201dbdaa6a873629c5cc28bffa9ab7d6039f3b73e9863c2: Status 404 returned error can't find the container with id f325ddb885b0f5b8f201dbdaa6a873629c5cc28bffa9ab7d6039f3b73e9863c2 Sep 16 14:09:25.270450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:25.270404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f5cb57f5-kldn8" event={"ID":"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec","Type":"ContainerStarted","Data":"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036"} Sep 16 14:09:25.270649 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:25.270469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f5cb57f5-kldn8" event={"ID":"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec","Type":"ContainerStarted","Data":"f325ddb885b0f5b8f201dbdaa6a873629c5cc28bffa9ab7d6039f3b73e9863c2"} Sep 16 14:09:25.301129 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:25.301076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9f5cb57f5-kldn8" podStartSLOduration=2.301060479 podStartE2EDuration="2.301060479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:25.298338407 +0000 UTC m=+95.875699560" watchObservedRunningTime="2026-09-16 14:09:25.301060479 +0000 UTC m=+95.878421631" Sep 16 14:09:28.121085 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:28.121052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5cgs2" Sep 16 14:09:28.584405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:28.584369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:28.729805 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:28.729765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:29.144168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:29.144136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:29.242393 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:29.242366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:31.700656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:31.700627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:31.771869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:31.771834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:34.233806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:34.233764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:34.234228 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:34.233836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:34.238717 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:34.238686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:34.313691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:34.313661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:09:34.910571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:34.910535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:35.060251 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:35.060214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:35.525112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:35.525078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:35.655718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:35.655687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:35.759350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:35.759309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:37.328173 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:37.328137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:37.409539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:37.409502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:44.641459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:44.641420 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69d6c6bd59-8fk4j" podUID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" containerName="console" containerID="cri-o://eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e" gracePeriod=15 Sep 16 14:09:44.881570 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:44.881550 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:45.019685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.019655 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wxfp\" (UniqueName: \"kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.019685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.019698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.019913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.019729 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.019913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.019749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.020021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.019994 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.020087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.020071 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config\") pod \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\" (UID: \"ed90a5e6-9d39-4aff-96d0-826a563f64fc\") " Sep 16 14:09:45.020217 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.020180 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:45.020217 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.020194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config" (OuterVolumeSpecName: "console-config") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:45.020813 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.020787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:45.021362 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.021328 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.021496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.021482 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-oauth-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.021641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.021591 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a5e6-9d39-4aff-96d0-826a563f64fc-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.024649 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.022867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp" (OuterVolumeSpecName: "kube-api-access-6wxfp") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "kube-api-access-6wxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:45.024649 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.023248 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:45.027650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.027628 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ed90a5e6-9d39-4aff-96d0-826a563f64fc" (UID: "ed90a5e6-9d39-4aff-96d0-826a563f64fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:45.122298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.122264 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.122298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.122291 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed90a5e6-9d39-4aff-96d0-826a563f64fc-console-oauth-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.122298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.122300 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxfp\" (UniqueName: \"kubernetes.io/projected/ed90a5e6-9d39-4aff-96d0-826a563f64fc-kube-api-access-6wxfp\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.353087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.352998 2579 generic.go:356] "Generic (PLEG): container finished" podID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" containerID="eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e" exitCode=0 Sep 16 14:09:45.353087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.353060 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d6c6bd59-8fk4j" Sep 16 14:09:45.353256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.353093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d6c6bd59-8fk4j" event={"ID":"ed90a5e6-9d39-4aff-96d0-826a563f64fc","Type":"ContainerDied","Data":"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e"} Sep 16 14:09:45.353256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.353162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d6c6bd59-8fk4j" event={"ID":"ed90a5e6-9d39-4aff-96d0-826a563f64fc","Type":"ContainerDied","Data":"ac8cd7ccbe98e61476afecf816d49a99937aea08c24733b036ed331bdb1186f3"} Sep 16 14:09:45.353256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.353182 2579 scope.go:117] "RemoveContainer" containerID="eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e" Sep 16 14:09:45.362111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.362094 2579 scope.go:117] "RemoveContainer" containerID="eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e" Sep 16 14:09:45.362389 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:45.362371 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e\": container with ID starting with eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e not found: ID does not exist" containerID="eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e" Sep 16 14:09:45.362435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.362396 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e"} err="failed to get container status \"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e\": rpc error: code = NotFound desc = could not find container \"eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e\": container with ID starting with eca74956bcfe5ac3fcb264d5965e2aa00a2f355cbb955a15ed76ed1141007e0e not found: ID does not exist" Sep 16 14:09:45.377502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.377478 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:45.394542 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.394517 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d6c6bd59-8fk4j"] Sep 16 14:09:45.815533 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:45.815502 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" path="/var/lib/kubelet/pods/ed90a5e6-9d39-4aff-96d0-826a563f64fc/volumes" Sep 16 14:09:46.358659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:46.358623 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac63b38a-300e-483e-94b4-06db8633505f" containerID="02f3ea42168701d28d8260bb1469e4d1f4c00b5fdfc5cd5e39d1edaac47c2a78" exitCode=0 Sep 16 14:09:46.358827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:46.358700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-z6mkc" event={"ID":"ac63b38a-300e-483e-94b4-06db8633505f","Type":"ContainerDied","Data":"02f3ea42168701d28d8260bb1469e4d1f4c00b5fdfc5cd5e39d1edaac47c2a78"} Sep 16 14:09:46.359056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:46.359037 2579 scope.go:117] "RemoveContainer" containerID="02f3ea42168701d28d8260bb1469e4d1f4c00b5fdfc5cd5e39d1edaac47c2a78" Sep 16 14:09:47.364893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.364859 2579 generic.go:356] "Generic (PLEG): container finished" podID="03e3a050-8164-4248-b342-549e1ebaa9f8" containerID="2e5280e75eca9b08d66592e27887cbff7a6dde97e8cf9cf263c674b5c4bc558c" exitCode=0 Sep 16 14:09:47.365308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.364931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" event={"ID":"03e3a050-8164-4248-b342-549e1ebaa9f8","Type":"ContainerDied","Data":"2e5280e75eca9b08d66592e27887cbff7a6dde97e8cf9cf263c674b5c4bc558c"} Sep 16 14:09:47.365308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.365278 2579 scope.go:117] "RemoveContainer" containerID="2e5280e75eca9b08d66592e27887cbff7a6dde97e8cf9cf263c674b5c4bc558c" Sep 16 14:09:47.369251 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.369232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-z6mkc" event={"ID":"ac63b38a-300e-483e-94b4-06db8633505f","Type":"ContainerStarted","Data":"cd1f72b7a43e36d70905b63a78b15a279f1eac0d5179eedbc1d8be1a3efc50ef"} Sep 16 14:09:47.373545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.373516 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" podUID="6f786f66-bf43-47a2-a5a5-c162fe54b013" containerName="registry" containerID="cri-o://66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964" gracePeriod=30 Sep 16 14:09:47.614536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.614508 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:09:47.743232 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2mx8\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743346 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743702 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") pod \"6f786f66-bf43-47a2-a5a5-c162fe54b013\" (UID: \"6f786f66-bf43-47a2-a5a5-c162fe54b013\") " Sep 16 14:09:47.743834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:47.743955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.743940 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-trusted-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.744089 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.744035 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:47.746372 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.746339 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:47.746478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.746378 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:47.746539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.746481 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8" (OuterVolumeSpecName: "kube-api-access-z2mx8") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "kube-api-access-z2mx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:47.746624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.746581 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:47.746688 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.746665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:47.758713 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.758686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6f786f66-bf43-47a2-a5a5-c162fe54b013" (UID: "6f786f66-bf43-47a2-a5a5-c162fe54b013"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:47.844571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844540 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f786f66-bf43-47a2-a5a5-c162fe54b013-ca-trust-extracted\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844563 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-installation-pull-secrets\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844572 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844583 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-bound-sa-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844591 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mx8\" (UniqueName: \"kubernetes.io/projected/6f786f66-bf43-47a2-a5a5-c162fe54b013-kube-api-access-z2mx8\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844601 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f786f66-bf43-47a2-a5a5-c162fe54b013-image-registry-private-configuration\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:47.844756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:47.844622 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f786f66-bf43-47a2-a5a5-c162fe54b013-registry-certificates\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:09:48.375485 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.375451 2579 generic.go:356] "Generic (PLEG): container finished" podID="6f786f66-bf43-47a2-a5a5-c162fe54b013" containerID="66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964" exitCode=0 Sep 16 14:09:48.375974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.375510 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" Sep 16 14:09:48.375974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.375538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" event={"ID":"6f786f66-bf43-47a2-a5a5-c162fe54b013","Type":"ContainerDied","Data":"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964"} Sep 16 14:09:48.375974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.375597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc95d58db-9c64m" event={"ID":"6f786f66-bf43-47a2-a5a5-c162fe54b013","Type":"ContainerDied","Data":"f97ccacef2e32a94120bdebd2cfb32eada2924ba2ea2987679b6dc5c55f8a632"} Sep 16 14:09:48.375974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.375642 2579 scope.go:117] "RemoveContainer" containerID="66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964" Sep 16 14:09:48.377803 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.377782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5nlgw" event={"ID":"03e3a050-8164-4248-b342-549e1ebaa9f8","Type":"ContainerStarted","Data":"1f6503171e5ab96b5d2a7d0338f1669d1e3f1cae5959c720f9d642ad72315348"} Sep 16 14:09:48.386130 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.386114 2579 scope.go:117] "RemoveContainer" containerID="66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964" Sep 16 14:09:48.386406 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:09:48.386388 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964\": container with ID starting with 66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964 not found: ID does not exist" containerID="66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964" Sep 16 14:09:48.386475 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.386414 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964"} err="failed to get container status \"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964\": rpc error: code = NotFound desc = could not find container \"66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964\": container with ID starting with 66630e8ba3d1b537fbc9471fa546c1f97924017a391db98ea8077eec0b52a964 not found: ID does not exist" Sep 16 14:09:48.422822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.422785 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:09:48.425770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:48.425744 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dc95d58db-9c64m"] Sep 16 14:09:49.816800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:49.816768 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f786f66-bf43-47a2-a5a5-c162fe54b013" path="/var/lib/kubelet/pods/6f786f66-bf43-47a2-a5a5-c162fe54b013/volumes" Sep 16 14:09:52.302422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:52.302388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:52.470217 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:52.470185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:57.414354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:57.414317 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe825bb-5841-464d-9dd4-1b41c72c77a9" containerID="6f50b59a4efee24cf56734878cef6d515369452e91601a96caf9ac5e133ea12b" exitCode=0 Sep 16 14:09:57.414834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:57.414390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" event={"ID":"efe825bb-5841-464d-9dd4-1b41c72c77a9","Type":"ContainerDied","Data":"6f50b59a4efee24cf56734878cef6d515369452e91601a96caf9ac5e133ea12b"} Sep 16 14:09:57.414834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:57.414811 2579 scope.go:117] "RemoveContainer" containerID="6f50b59a4efee24cf56734878cef6d515369452e91601a96caf9ac5e133ea12b" Sep 16 14:09:58.421044 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:09:58.421011 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-skt6c" event={"ID":"efe825bb-5841-464d-9dd4-1b41c72c77a9","Type":"ContainerStarted","Data":"bbfbf02fa0bb00e06bd14044c573c4b605da19255fe0a11d27a263694dc41028"} Sep 16 14:10:12.611359 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:12.611322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:17.880422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:17.880350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:32.449986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:32.449945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:32.732225 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:32.732139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:41.028631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.028580 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:10:41.029092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029034 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" containerName="console" Sep 16 14:10:41.029092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029051 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" containerName="console" Sep 16 14:10:41.029092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029070 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f786f66-bf43-47a2-a5a5-c162fe54b013" containerName="registry" Sep 16 14:10:41.029092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029079 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f786f66-bf43-47a2-a5a5-c162fe54b013" containerName="registry" Sep 16 14:10:41.029226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029189 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed90a5e6-9d39-4aff-96d0-826a563f64fc" containerName="console" Sep 16 14:10:41.029226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.029202 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f786f66-bf43-47a2-a5a5-c162fe54b013" containerName="registry" Sep 16 14:10:41.032209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.032193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051707 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzwr\" (UniqueName: \"kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.051928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.051870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.061051 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.061022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:10:41.137917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.137887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:41.153284 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153646 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.153646 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.153456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzwr\" (UniqueName: \"kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.154059 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.154034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.154227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.154208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.154287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.154268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.154446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.154424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.156023 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.155994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.156100 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.156082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.166080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.166057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzwr\" (UniqueName: \"kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr\") pod \"console-7dc8f4b6cb-5nc8x\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.236249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.236213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:41.342179 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.342080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:41.621720 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:41.621636 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:10:41.625279 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:10:41.625244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cdead9_1444_46cd_a673_e6b4da0e775e.slice/crio-7e6680c1d7f728e95f44e9853d78b1c6934a54377b4e947ddfe446522ac9bcdc WatchSource:0}: Error finding container 7e6680c1d7f728e95f44e9853d78b1c6934a54377b4e947ddfe446522ac9bcdc: Status 404 returned error can't find the container with id 7e6680c1d7f728e95f44e9853d78b1c6934a54377b4e947ddfe446522ac9bcdc Sep 16 14:10:42.593904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:42.593871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc8f4b6cb-5nc8x" event={"ID":"b8cdead9-1444-46cd-a673-e6b4da0e775e","Type":"ContainerStarted","Data":"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5"} Sep 16 14:10:42.594352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:42.593924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc8f4b6cb-5nc8x" event={"ID":"b8cdead9-1444-46cd-a673-e6b4da0e775e","Type":"ContainerStarted","Data":"7e6680c1d7f728e95f44e9853d78b1c6934a54377b4e947ddfe446522ac9bcdc"} Sep 16 14:10:51.342646 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.342576 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:51.343146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.342687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:51.347494 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.347468 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:51.370677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.370625 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc8f4b6cb-5nc8x" podStartSLOduration=11.370590668 podStartE2EDuration="11.370590668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:10:42.638734865 +0000 UTC m=+173.216096016" watchObservedRunningTime="2026-09-16 14:10:51.370590668 +0000 UTC m=+181.947951820" Sep 16 14:10:51.632271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.632189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:10:51.680024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.679993 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:10:51.794489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.794454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:51.896459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:51.896368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:52.734983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:10:52.734945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:06.852976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:06.852933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:16.703683 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:16.703644 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9f5cb57f5-kldn8" podUID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" containerName="console" containerID="cri-o://314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036" gracePeriod=15 Sep 16 14:11:16.944802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:16.944764 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:11:17.039086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039015 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039057 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039104 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6njmn\" (UniqueName: \"kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039140 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039191 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert\") pod \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\" (UID: \"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec\") " Sep 16 14:11:17.039651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:11:17.040017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.039986 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config" (OuterVolumeSpecName: "console-config") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:11:17.040658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.040584 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:11:17.042025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.040869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:11:17.042347 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.042305 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn" (OuterVolumeSpecName: "kube-api-access-6njmn") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "kube-api-access-6njmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:11:17.042544 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.042505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:17.042656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.042576 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" (UID: "d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:17.141231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141205 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-trusted-ca-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141231 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-oauth-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141244 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141256 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6njmn\" (UniqueName: \"kubernetes.io/projected/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-kube-api-access-6njmn\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141267 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141277 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-oauth-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.141407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.141289 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec-console-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:11:17.727774 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.727731 2579 generic.go:356] "Generic (PLEG): container finished" podID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" containerID="314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036" exitCode=0 Sep 16 14:11:17.728165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.727792 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f5cb57f5-kldn8" Sep 16 14:11:17.728165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.727816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f5cb57f5-kldn8" event={"ID":"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec","Type":"ContainerDied","Data":"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036"} Sep 16 14:11:17.728165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.727890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f5cb57f5-kldn8" event={"ID":"d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec","Type":"ContainerDied","Data":"f325ddb885b0f5b8f201dbdaa6a873629c5cc28bffa9ab7d6039f3b73e9863c2"} Sep 16 14:11:17.728165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.727911 2579 scope.go:117] "RemoveContainer" containerID="314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036" Sep 16 14:11:17.736574 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.736558 2579 scope.go:117] "RemoveContainer" containerID="314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036" Sep 16 14:11:17.736892 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:11:17.736872 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036\": container with ID starting with 314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036 not found: ID does not exist" containerID="314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036" Sep 16 14:11:17.736957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.736899 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036"} err="failed to get container status \"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036\": rpc error: code = NotFound desc = could not find container \"314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036\": container with ID starting with 314485aa09932ada321eaffd7607c35f7fdceb98f934664dda78b3b80552c036 not found: ID does not exist" Sep 16 14:11:17.762259 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.762239 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:11:17.769013 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.768992 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9f5cb57f5-kldn8"] Sep 16 14:11:17.815954 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:17.815928 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" path="/var/lib/kubelet/pods/d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec/volumes" Sep 16 14:11:18.293113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:18.293079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:18.433293 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:18.433263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:47.784254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:47.784220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:47.884685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:47.884643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:56.747567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.747531 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:11:56.748075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.747875 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" containerName="console" Sep 16 14:11:56.748075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.747889 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" containerName="console" Sep 16 14:11:56.748075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.747936 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d56b9b-0403-44c3-a2a8-2d3a9250c9ec" containerName="console" Sep 16 14:11:56.752218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.752197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.755571 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:11:56.755536 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-7bb7f7fb9-v26gc\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.773992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.773967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:11:56.846238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4lp\" (UniqueName: \"kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.846447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.846370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4lp\" (UniqueName: \"kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.947945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.947919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.948084 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.948067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.948220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.948195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.948451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.948426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.949884 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.949861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.950027 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.950009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:56.956188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:56.956160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4lp\" (UniqueName: \"kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp\") pod \"console-7bb7f7fb9-v26gc\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:57.064751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.064693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:11:57.339023 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.338992 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:11:57.341522 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:11:57.341501 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf809fdf0_46a6_45ee_975f_eaabed778bec.slice/crio-686313e6b07e52bcdda46f9fba8df4c1c51aef63f01d836e467a0cfb44a97a00 WatchSource:0}: Error finding container 686313e6b07e52bcdda46f9fba8df4c1c51aef63f01d836e467a0cfb44a97a00: Status 404 returned error can't find the container with id 686313e6b07e52bcdda46f9fba8df4c1c51aef63f01d836e467a0cfb44a97a00 Sep 16 14:11:57.485160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.485123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:57.546360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.546321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:57.881967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.881934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb7f7fb9-v26gc" event={"ID":"f809fdf0-46a6-45ee-975f-eaabed778bec","Type":"ContainerStarted","Data":"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5"} Sep 16 14:11:57.882331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.881977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb7f7fb9-v26gc" event={"ID":"f809fdf0-46a6-45ee-975f-eaabed778bec","Type":"ContainerStarted","Data":"686313e6b07e52bcdda46f9fba8df4c1c51aef63f01d836e467a0cfb44a97a00"} Sep 16 14:11:57.902721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:57.902678 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb7f7fb9-v26gc" podStartSLOduration=1.902662733 podStartE2EDuration="1.902662733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:57.901132141 +0000 UTC m=+248.478493296" watchObservedRunningTime="2026-09-16 14:11:57.902662733 +0000 UTC m=+248.480023885" Sep 16 14:11:59.723297 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:59.723258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:59.787470 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:11:59.787442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:07.065100 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.065068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:12:07.065565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.065122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:12:07.070034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.070014 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:12:07.238213 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.238177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:07.339080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.338982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:07.395092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.395047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:07.925162 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.925132 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:12:07.974551 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:07.974509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:12:08.084535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:08.084497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:08.181519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:08.181426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:08.971098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:08.971058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:32.995312 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:32.995247 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dc8f4b6cb-5nc8x" podUID="b8cdead9-1444-46cd-a673-e6b4da0e775e" containerName="console" containerID="cri-o://ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5" gracePeriod=15 Sep 16 14:12:33.228063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.228041 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:12:33.337786 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzwr\" (UniqueName: \"kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.337786 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337747 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.337786 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337763 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.338057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337799 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.338057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.338057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.338057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.337924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config\") pod \"b8cdead9-1444-46cd-a673-e6b4da0e775e\" (UID: \"b8cdead9-1444-46cd-a673-e6b4da0e775e\") " Sep 16 14:12:33.338306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.338283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config" (OuterVolumeSpecName: "console-config") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:12:33.338366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.338292 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:12:33.338366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.338311 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:12:33.338366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.338307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:12:33.340149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.340122 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:12:33.340149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.340139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr" (OuterVolumeSpecName: "kube-api-access-4wzwr") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "kube-api-access-4wzwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:12:33.340293 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.340161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8cdead9-1444-46cd-a673-e6b4da0e775e" (UID: "b8cdead9-1444-46cd-a673-e6b4da0e775e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:12:33.439299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439255 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-oauth-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439291 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzwr\" (UniqueName: \"kubernetes.io/projected/b8cdead9-1444-46cd-a673-e6b4da0e775e-kube-api-access-4wzwr\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439305 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-trusted-ca-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439319 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439333 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cdead9-1444-46cd-a673-e6b4da0e775e-console-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439344 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-oauth-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:33.439529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:33.439354 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cdead9-1444-46cd-a673-e6b4da0e775e-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:12:34.022195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.022162 2579 generic.go:356] "Generic (PLEG): container finished" podID="b8cdead9-1444-46cd-a673-e6b4da0e775e" containerID="ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5" exitCode=0 Sep 16 14:12:34.022643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.022204 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc8f4b6cb-5nc8x" Sep 16 14:12:34.022643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.022213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc8f4b6cb-5nc8x" event={"ID":"b8cdead9-1444-46cd-a673-e6b4da0e775e","Type":"ContainerDied","Data":"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5"} Sep 16 14:12:34.022643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.022252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc8f4b6cb-5nc8x" event={"ID":"b8cdead9-1444-46cd-a673-e6b4da0e775e","Type":"ContainerDied","Data":"7e6680c1d7f728e95f44e9853d78b1c6934a54377b4e947ddfe446522ac9bcdc"} Sep 16 14:12:34.022643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.022271 2579 scope.go:117] "RemoveContainer" containerID="ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5" Sep 16 14:12:34.030834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.030817 2579 scope.go:117] "RemoveContainer" containerID="ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5" Sep 16 14:12:34.031119 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:12:34.031090 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5\": container with ID starting with ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5 not found: ID does not exist" containerID="ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5" Sep 16 14:12:34.031184 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.031127 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5"} err="failed to get container status \"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5\": rpc error: code = NotFound desc = could not find container \"ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5\": container with ID starting with ec8e96dfbddd5dd99c41ed39ed8e51b9c29d41b918bc79cd880d5edc60bda6e5 not found: ID does not exist" Sep 16 14:12:34.043924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.043900 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:12:34.048770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:34.048751 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc8f4b6cb-5nc8x"] Sep 16 14:12:35.816357 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:35.816318 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8cdead9-1444-46cd-a673-e6b4da0e775e" path="/var/lib/kubelet/pods/b8cdead9-1444-46cd-a673-e6b4da0e775e/volumes" Sep 16 14:12:47.950087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:47.950053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:48.044129 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.044090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:48.921909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.921881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc"] Sep 16 14:12:48.922206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.922191 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8cdead9-1444-46cd-a673-e6b4da0e775e" containerName="console" Sep 16 14:12:48.922279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.922208 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8cdead9-1444-46cd-a673-e6b4da0e775e" containerName="console" Sep 16 14:12:48.922279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.922274 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8cdead9-1444-46cd-a673-e6b4da0e775e" containerName="console" Sep 16 14:12:48.925390 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.925374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:48.928100 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.928075 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 14:12:48.929030 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.929012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 14:12:48.929151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.929126 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5wbqx\"" Sep 16 14:12:48.938212 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.938191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc"] Sep 16 14:12:48.955368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.955324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:48.956037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.955391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:48.956037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:48.955476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzc2\" (UniqueName: \"kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.056021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.055989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.056143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.056034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.056143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.056076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzc2\" (UniqueName: \"kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.056389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.056369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.056430 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.056388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.065409 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.065380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzc2\" (UniqueName: \"kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.234823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.234782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:12:49.509637 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.509538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc"] Sep 16 14:12:49.511864 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:12:49.511837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de3fef4_4f96_49c3_ac19_eda1c172a4cf.slice/crio-1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663 WatchSource:0}: Error finding container 1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663: Status 404 returned error can't find the container with id 1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663 Sep 16 14:12:49.755691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.755661 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:12:49.757352 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.757334 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:12:49.767124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:49.767058 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 14:12:50.081928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:50.081839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerStarted","Data":"1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663"} Sep 16 14:12:56.109999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:56.109918 2579 generic.go:356] "Generic (PLEG): container finished" podID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerID="268bce4b5a471e02eb3d679e9bfbda7b31d6f1408fbe232af99472d8b3cadd89" exitCode=0 Sep 16 14:12:56.109999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:56.109971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerDied","Data":"268bce4b5a471e02eb3d679e9bfbda7b31d6f1408fbe232af99472d8b3cadd89"} Sep 16 14:12:56.110960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:56.110943 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:12:59.124468 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:59.124435 2579 generic.go:356] "Generic (PLEG): container finished" podID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerID="f7b3aaf7a33cb096f35f53c4394a7bf25fbe0f947aff47012ae1e85725108c5c" exitCode=0 Sep 16 14:12:59.124898 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:59.124491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerDied","Data":"f7b3aaf7a33cb096f35f53c4394a7bf25fbe0f947aff47012ae1e85725108c5c"} Sep 16 14:12:59.297276 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:12:59.297242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:05.155115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:05.155076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerStarted","Data":"748d5daa1cc646b1db5ea480f42e053b9fbc22ceaa9b72d82ffe2e89fc27105a"} Sep 16 14:13:05.177833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:05.177772 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" podStartSLOduration=2.978505672 podStartE2EDuration="17.177754517s" totalImagesPullingTime="14.199248845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:05.176704927 +0000 UTC m=+315.754066090" watchObservedRunningTime="2026-09-16 14:13:05.177754517 +0000 UTC m=+315.755115669" Sep 16 14:13:06.160655 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:06.160622 2579 generic.go:356] "Generic (PLEG): container finished" podID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerID="748d5daa1cc646b1db5ea480f42e053b9fbc22ceaa9b72d82ffe2e89fc27105a" exitCode=0 Sep 16 14:13:06.161041 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:06.160704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerDied","Data":"748d5daa1cc646b1db5ea480f42e053b9fbc22ceaa9b72d82ffe2e89fc27105a"} Sep 16 14:13:07.290336 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.290309 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:13:07.407332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.407296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle\") pod \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " Sep 16 14:13:07.407332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.407346 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxzc2\" (UniqueName: \"kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2\") pod \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " Sep 16 14:13:07.407555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.407386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util\") pod \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\" (UID: \"8de3fef4-4f96-49c3-ac19-eda1c172a4cf\") " Sep 16 14:13:07.408242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.408155 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle" (OuterVolumeSpecName: "bundle") pod "8de3fef4-4f96-49c3-ac19-eda1c172a4cf" (UID: "8de3fef4-4f96-49c3-ac19-eda1c172a4cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:07.409806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.409782 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2" (OuterVolumeSpecName: "kube-api-access-nxzc2") pod "8de3fef4-4f96-49c3-ac19-eda1c172a4cf" (UID: "8de3fef4-4f96-49c3-ac19-eda1c172a4cf"). InnerVolumeSpecName "kube-api-access-nxzc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:07.413845 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.413822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util" (OuterVolumeSpecName: "util") pod "8de3fef4-4f96-49c3-ac19-eda1c172a4cf" (UID: "8de3fef4-4f96-49c3-ac19-eda1c172a4cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:07.508311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.508272 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:07.508311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.508301 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:07.508311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:07.508312 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzc2\" (UniqueName: \"kubernetes.io/projected/8de3fef4-4f96-49c3-ac19-eda1c172a4cf-kube-api-access-nxzc2\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:08.169335 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:08.169259 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" Sep 16 14:13:08.169487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:08.169259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b62pc" event={"ID":"8de3fef4-4f96-49c3-ac19-eda1c172a4cf","Type":"ContainerDied","Data":"1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663"} Sep 16 14:13:08.169487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:08.169385 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1469cb217485a8888d8677ff720160e1be6cc550dfa736e68a803a2b512e7663" Sep 16 14:13:10.041395 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:10.041361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:10.995576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:10.995536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:12.054361 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054327 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7"] Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054650 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="pull" Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054663 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="pull" Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054674 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="extract" Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054679 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="extract" Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054704 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="util" Sep 16 14:13:12.054746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054709 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="util" Sep 16 14:13:12.054966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.054764 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de3fef4-4f96-49c3-ac19-eda1c172a4cf" containerName="extract" Sep 16 14:13:12.064800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.064778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.067571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.067547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gbj76\"" Sep 16 14:13:12.067681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.067588 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 14:13:12.067855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.067842 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 14:13:12.073009 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.072988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7"] Sep 16 14:13:12.148995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.148961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8b56ec-fb55-47c2-8071-375d023174f8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.149154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.149009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2q7\" (UniqueName: \"kubernetes.io/projected/6b8b56ec-fb55-47c2-8071-375d023174f8-kube-api-access-7t2q7\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.249628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.249567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8b56ec-fb55-47c2-8071-375d023174f8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.249787 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.249654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2q7\" (UniqueName: \"kubernetes.io/projected/6b8b56ec-fb55-47c2-8071-375d023174f8-kube-api-access-7t2q7\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.249960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.249940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8b56ec-fb55-47c2-8071-375d023174f8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.258943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.258918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2q7\" (UniqueName: \"kubernetes.io/projected/6b8b56ec-fb55-47c2-8071-375d023174f8-kube-api-access-7t2q7\") pod \"cert-manager-operator-controller-manager-55c444bf86-grbt7\" (UID: \"6b8b56ec-fb55-47c2-8071-375d023174f8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.374330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.374261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" Sep 16 14:13:12.677925 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:12.677903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7"] Sep 16 14:13:13.191659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:13.191622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" event={"ID":"6b8b56ec-fb55-47c2-8071-375d023174f8","Type":"ContainerStarted","Data":"d27590356812525b0aebc744b7e1f315b85db02812010bf9cd1b5ac3987ed1b5"} Sep 16 14:13:13.686540 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:13.686494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:13.785062 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:13.785028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:15.203538 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.203476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" event={"ID":"6b8b56ec-fb55-47c2-8071-375d023174f8","Type":"ContainerStarted","Data":"f5435b01f170fd6b01e07c763ad2277784683daa9c1834f1fa3dbdac72a3476b"} Sep 16 14:13:15.231082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.231032 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grbt7" podStartSLOduration=1.1359477199999999 podStartE2EDuration="3.231014313s" totalImagesPullingTime="2.095066593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:15.229960126 +0000 UTC m=+325.807321288" watchObservedRunningTime="2026-09-16 14:13:15.231014313 +0000 UTC m=+325.808375468" Sep 16 14:13:15.568187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.568084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:15.705807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.705751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:15.817174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.817147 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pp4zt"] Sep 16 14:13:15.834481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.834422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:15.845007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.844987 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 14:13:15.845120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.845014 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 14:13:15.874914 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.874889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pp4zt"] Sep 16 14:13:15.937918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.937883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:15.983182 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.983159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:15.983316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:15.983187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xkw\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-kube-api-access-26xkw\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.083869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.083841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.083869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.083870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xkw\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-kube-api-access-26xkw\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.108960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.108893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.113216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.113196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xkw\" (UniqueName: \"kubernetes.io/projected/9c8fc6aa-a94e-484a-899a-b77a8e68f440-kube-api-access-26xkw\") pod \"cert-manager-webhook-6486ffd997-pp4zt\" (UID: \"9c8fc6aa-a94e-484a-899a-b77a8e68f440\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.157204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.157181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:16.434668 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.434643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pp4zt"] Sep 16 14:13:16.437005 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:16.436972 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c8fc6aa_a94e_484a_899a_b77a8e68f440.slice/crio-767daf4fa42695a8743f7f62cc47acebf4fbe00ef903f9f68a840898b9863389 WatchSource:0}: Error finding container 767daf4fa42695a8743f7f62cc47acebf4fbe00ef903f9f68a840898b9863389: Status 404 returned error can't find the container with id 767daf4fa42695a8743f7f62cc47acebf4fbe00ef903f9f68a840898b9863389 Sep 16 14:13:16.592322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.592287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv"] Sep 16 14:13:16.596770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.596752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.600377 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.600357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 14:13:16.601374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.601357 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5wbqx\"" Sep 16 14:13:16.610148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.610131 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 14:13:16.615708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.615686 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv"] Sep 16 14:13:16.690526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.690445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.690526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.690485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.690526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.690518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkw4\" (UniqueName: \"kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.791374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.791340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.791545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.791389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkw4\" (UniqueName: \"kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.791545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.791461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.791771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.791751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.791843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.791830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.801212 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.801193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkw4\" (UniqueName: \"kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:16.905582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:16.905550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:17.187666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:17.187634 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv"] Sep 16 14:13:17.188677 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:17.188647 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b5c932_afe5_4ec4_8dac_7e9f3b7a15f5.slice/crio-8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865 WatchSource:0}: Error finding container 8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865: Status 404 returned error can't find the container with id 8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865 Sep 16 14:13:17.215754 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:17.215720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" event={"ID":"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5","Type":"ContainerStarted","Data":"8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865"} Sep 16 14:13:17.216997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:17.216976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" event={"ID":"9c8fc6aa-a94e-484a-899a-b77a8e68f440","Type":"ContainerStarted","Data":"767daf4fa42695a8743f7f62cc47acebf4fbe00ef903f9f68a840898b9863389"} Sep 16 14:13:17.957157 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:17.957115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:18.156776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:18.156738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:18.224005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:18.223914 2579 generic.go:356] "Generic (PLEG): container finished" podID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerID="8ea9373cf8557965c24f6d82815d03f657cca34febc46499bd532fd1e15ba95a" exitCode=0 Sep 16 14:13:18.224167 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:18.224015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" event={"ID":"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5","Type":"ContainerDied","Data":"8ea9373cf8557965c24f6d82815d03f657cca34febc46499bd532fd1e15ba95a"} Sep 16 14:13:20.236967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:20.236921 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" event={"ID":"9c8fc6aa-a94e-484a-899a-b77a8e68f440","Type":"ContainerStarted","Data":"b14c867dcf968be4c44b03d4240dc8bdeadb6433d04201e96bd8abe31f0c60da"} Sep 16 14:13:20.237348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:20.237059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:20.269238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:20.269190 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" podStartSLOduration=2.499069182 podStartE2EDuration="5.269173448s" totalImagesPullingTime="2.770104266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:20.267226954 +0000 UTC m=+330.844588105" watchObservedRunningTime="2026-09-16 14:13:20.269173448 +0000 UTC m=+330.846534600" Sep 16 14:13:20.839480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:20.839425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:20.902917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:20.902890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:23.254632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:23.254574 2579 generic.go:356] "Generic (PLEG): container finished" podID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerID="fa79934325defba0243c46164dd2a7b8474fd55548a259872958110513ed72c8" exitCode=0 Sep 16 14:13:23.255018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:23.254722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" event={"ID":"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5","Type":"ContainerDied","Data":"fa79934325defba0243c46164dd2a7b8474fd55548a259872958110513ed72c8"} Sep 16 14:13:24.261461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:24.261427 2579 generic.go:356] "Generic (PLEG): container finished" podID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerID="189532693f45d0b48f5928bd40cf6ea67207789cc6b8acc8105ebb492023e6cc" exitCode=0 Sep 16 14:13:24.261833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:24.261505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" event={"ID":"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5","Type":"ContainerDied","Data":"189532693f45d0b48f5928bd40cf6ea67207789cc6b8acc8105ebb492023e6cc"} Sep 16 14:13:25.199250 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.199213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:25.383443 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.383421 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:25.471029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.470943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util\") pod \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " Sep 16 14:13:25.471029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.470989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle\") pod \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " Sep 16 14:13:25.471029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.471014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpkw4\" (UniqueName: \"kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4\") pod \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\" (UID: \"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5\") " Sep 16 14:13:25.471374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.471340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle" (OuterVolumeSpecName: "bundle") pod "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" (UID: "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:25.473219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.473194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4" (OuterVolumeSpecName: "kube-api-access-jpkw4") pod "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" (UID: "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5"). InnerVolumeSpecName "kube-api-access-jpkw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:25.475776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.475751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util" (OuterVolumeSpecName: "util") pod "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" (UID: "53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:25.572287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.572243 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:25.572287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.572279 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkw4\" (UniqueName: \"kubernetes.io/projected/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-kube-api-access-jpkw4\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:25.572287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:25.572293 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:26.243804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:26.243778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pp4zt" Sep 16 14:13:26.272762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:26.272732 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" Sep 16 14:13:26.272916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:26.272771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg5gbv" event={"ID":"53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5","Type":"ContainerDied","Data":"8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865"} Sep 16 14:13:26.272916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:26.272824 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b614cf618694cc0ee14ed887567c8cd63e3618fc81b733b048f1405afba6865" Sep 16 14:13:27.051658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051623 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5rmw"] Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051915 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="util" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="util" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051946 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="pull" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="pull" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051959 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="extract" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.051964 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="extract" Sep 16 14:13:27.052095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.052013 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b5c932-afe5-4ec4-8dac-7e9f3b7a15f5" containerName="extract" Sep 16 14:13:27.054705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.054689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.059980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.059955 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4nhll\"" Sep 16 14:13:27.080152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.080126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5rmw"] Sep 16 14:13:27.185224 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.185194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lx55\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-kube-api-access-8lx55\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.185360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.185246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.286067 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.286036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.286210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.286082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lx55\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-kube-api-access-8lx55\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.308384 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.308318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.309354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.309337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lx55\" (UniqueName: \"kubernetes.io/projected/f8d896fe-cd30-42b0-9591-ad241454cd87-kube-api-access-8lx55\") pod \"cert-manager-55b4d954d5-f5rmw\" (UID: \"f8d896fe-cd30-42b0-9591-ad241454cd87\") " pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.363322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.363297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f5rmw" Sep 16 14:13:27.435499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.435145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:27.565479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.565399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:27.646586 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:27.646559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5rmw"] Sep 16 14:13:27.649326 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:27.649292 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8d896fe_cd30_42b0_9591_ad241454cd87.slice/crio-1c6e2674004c6e1cf69a747f1553874a5bc1cf5dd9246bc9c3bbcc93fa40cd01 WatchSource:0}: Error finding container 1c6e2674004c6e1cf69a747f1553874a5bc1cf5dd9246bc9c3bbcc93fa40cd01: Status 404 returned error can't find the container with id 1c6e2674004c6e1cf69a747f1553874a5bc1cf5dd9246bc9c3bbcc93fa40cd01 Sep 16 14:13:28.282566 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:28.282528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f5rmw" event={"ID":"f8d896fe-cd30-42b0-9591-ad241454cd87","Type":"ContainerStarted","Data":"3e5870c5a9c04e94750938dc047552667390e5e8509cfdf1cb891c27112375d1"} Sep 16 14:13:28.282566 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:28.282567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f5rmw" event={"ID":"f8d896fe-cd30-42b0-9591-ad241454cd87","Type":"ContainerStarted","Data":"1c6e2674004c6e1cf69a747f1553874a5bc1cf5dd9246bc9c3bbcc93fa40cd01"} Sep 16 14:13:28.300628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:28.300568 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f5rmw" podStartSLOduration=1.300553286 podStartE2EDuration="1.300553286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:28.299532022 +0000 UTC m=+338.876893174" watchObservedRunningTime="2026-09-16 14:13:28.300553286 +0000 UTC m=+338.877914437" Sep 16 14:13:29.081187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:29.081148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:29.175098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:29.175065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:35.307981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:35.307944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:36.662715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.662679 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh"] Sep 16 14:13:36.668536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.668520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.671038 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.671012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 14:13:36.671166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.671016 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5wbqx\"" Sep 16 14:13:36.671166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.671016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 14:13:36.674815 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.674794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh"] Sep 16 14:13:36.756497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.756468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drz5j\" (UniqueName: \"kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.756695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.756525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.756695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.756567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.857466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.857434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.857466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.857473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.857698 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.857517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drz5j\" (UniqueName: \"kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.857818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.857799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.857914 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.857897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.866080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.866058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drz5j\" (UniqueName: \"kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:36.980093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:36.980069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:37.255054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:37.254971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh"] Sep 16 14:13:37.257257 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:37.257226 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42277844_b9af_4b2c_a8d5_6b99a69c921a.slice/crio-63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453 WatchSource:0}: Error finding container 63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453: Status 404 returned error can't find the container with id 63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453 Sep 16 14:13:37.323216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:37.323175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" event={"ID":"42277844-b9af-4b2c-a8d5-6b99a69c921a","Type":"ContainerStarted","Data":"63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453"} Sep 16 14:13:37.380124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:37.380097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:37.484435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:37.484399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:38.328896 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:38.328862 2579 generic.go:356] "Generic (PLEG): container finished" podID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerID="c3ab8c5ad4f05b036562a769a54c6c73befe32859c4cd21646ac51f6c3bb0aea" exitCode=0 Sep 16 14:13:38.329272 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:38.328944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" event={"ID":"42277844-b9af-4b2c-a8d5-6b99a69c921a","Type":"ContainerDied","Data":"c3ab8c5ad4f05b036562a769a54c6c73befe32859c4cd21646ac51f6c3bb0aea"} Sep 16 14:13:39.335024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:39.334937 2579 generic.go:356] "Generic (PLEG): container finished" podID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerID="4eff3489b055e9d50d49954c43ecf7f10dce0ca9cfddcf498c349618fcc291ee" exitCode=0 Sep 16 14:13:39.335389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:39.335017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" event={"ID":"42277844-b9af-4b2c-a8d5-6b99a69c921a","Type":"ContainerDied","Data":"4eff3489b055e9d50d49954c43ecf7f10dce0ca9cfddcf498c349618fcc291ee"} Sep 16 14:13:40.341920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:40.341889 2579 generic.go:356] "Generic (PLEG): container finished" podID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerID="5b36ae4d5af94ed747caa89e9385f8cfd21567a46a58836a76db518065267a7a" exitCode=0 Sep 16 14:13:40.342327 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:40.341979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" event={"ID":"42277844-b9af-4b2c-a8d5-6b99a69c921a","Type":"ContainerDied","Data":"5b36ae4d5af94ed747caa89e9385f8cfd21567a46a58836a76db518065267a7a"} Sep 16 14:13:41.472333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.472310 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:41.491055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.491028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drz5j\" (UniqueName: \"kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j\") pod \"42277844-b9af-4b2c-a8d5-6b99a69c921a\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " Sep 16 14:13:41.491270 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.491124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util\") pod \"42277844-b9af-4b2c-a8d5-6b99a69c921a\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " Sep 16 14:13:41.491656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.491631 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle\") pod \"42277844-b9af-4b2c-a8d5-6b99a69c921a\" (UID: \"42277844-b9af-4b2c-a8d5-6b99a69c921a\") " Sep 16 14:13:41.492520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.492476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle" (OuterVolumeSpecName: "bundle") pod "42277844-b9af-4b2c-a8d5-6b99a69c921a" (UID: "42277844-b9af-4b2c-a8d5-6b99a69c921a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:41.493819 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.493796 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j" (OuterVolumeSpecName: "kube-api-access-drz5j") pod "42277844-b9af-4b2c-a8d5-6b99a69c921a" (UID: "42277844-b9af-4b2c-a8d5-6b99a69c921a"). InnerVolumeSpecName "kube-api-access-drz5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:41.498865 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.498844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util" (OuterVolumeSpecName: "util") pod "42277844-b9af-4b2c-a8d5-6b99a69c921a" (UID: "42277844-b9af-4b2c-a8d5-6b99a69c921a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:13:41.592922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.592888 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drz5j\" (UniqueName: \"kubernetes.io/projected/42277844-b9af-4b2c-a8d5-6b99a69c921a-kube-api-access-drz5j\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:41.592922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.592916 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:41.592922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:41.592925 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42277844-b9af-4b2c-a8d5-6b99a69c921a-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:13:42.354163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:42.354127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" event={"ID":"42277844-b9af-4b2c-a8d5-6b99a69c921a","Type":"ContainerDied","Data":"63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453"} Sep 16 14:13:42.354163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:42.354154 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af65jrh" Sep 16 14:13:42.354364 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:42.354177 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c4b53df97b34f8ca423c186a380a9f1a6e03aba3131556966bf3b5fdbe1453" Sep 16 14:13:47.766727 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:47.766693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:47.879278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:47.879237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:50.623882 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:50.623844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:50.725871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:50.725834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:52.681987 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.681950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw"] Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682252 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="extract" Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682263 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="extract" Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682275 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="util" Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682281 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="util" Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682290 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="pull" Sep 16 14:13:52.682360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682295 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="pull" Sep 16 14:13:52.682543 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.682365 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42277844-b9af-4b2c-a8d5-6b99a69c921a" containerName="extract" Sep 16 14:13:52.688108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.688085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.690921 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.690901 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 14:13:52.691045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.690960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 14:13:52.691186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.691173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5wbqx\"" Sep 16 14:13:52.695786 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.695767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw"] Sep 16 14:13:52.783853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.783817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.784015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.783874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hgz\" (UniqueName: \"kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.784015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.783924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.886580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.885139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hgz\" (UniqueName: \"kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.886580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.885224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.886580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.885292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.886580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.885810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.886580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.886152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.895662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.895639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hgz\" (UniqueName: \"kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:52.998369 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:52.998292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:13:53.237113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:53.237081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:53.291395 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:53.291321 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw"] Sep 16 14:13:53.294063 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:53.294025 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3a17df_ae39_4e28_b395_31d96165ca45.slice/crio-3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d WatchSource:0}: Error finding container 3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d: Status 404 returned error can't find the container with id 3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d Sep 16 14:13:53.347355 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:53.347326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:53.404075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:53.404045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" event={"ID":"5e3a17df-ae39-4e28-b395-31d96165ca45","Type":"ContainerStarted","Data":"3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d"} Sep 16 14:13:53.867991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:53.867954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.130996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.130898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.256244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.256203 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6"] Sep 16 14:13:54.260442 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.260424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.267240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.267214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-plblj\"" Sep 16 14:13:54.267753 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.267682 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 14:13:54.268462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.268441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 14:13:54.268931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.268912 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 14:13:54.270751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.270732 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 14:13:54.278772 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.278748 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6"] Sep 16 14:13:54.296939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.296914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-manifests\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.297087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.296946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-tmp\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.297087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.297001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.297087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.297022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.297087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.297037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrzq\" (UniqueName: \"kubernetes.io/projected/c379fead-1cd2-4d3c-8327-e96957a5fb6b-kube-api-access-ttrzq\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrzq\" (UniqueName: \"kubernetes.io/projected/c379fead-1cd2-4d3c-8327-e96957a5fb6b-kube-api-access-ttrzq\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-manifests\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-tmp\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.398986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.398957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-manifests\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.400791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.400763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c379fead-1cd2-4d3c-8327-e96957a5fb6b-tmp\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.401060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.401043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.401108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.401097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c379fead-1cd2-4d3c-8327-e96957a5fb6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.410670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.410640 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerID="de9d9ae0ac114a9822a947edd2b57777140cf2d8e0bf0e307a2bd61dc3039315" exitCode=0 Sep 16 14:13:54.410763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.410684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" event={"ID":"5e3a17df-ae39-4e28-b395-31d96165ca45","Type":"ContainerDied","Data":"de9d9ae0ac114a9822a947edd2b57777140cf2d8e0bf0e307a2bd61dc3039315"} Sep 16 14:13:54.420139 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.420119 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrzq\" (UniqueName: \"kubernetes.io/projected/c379fead-1cd2-4d3c-8327-e96957a5fb6b-kube-api-access-ttrzq\") pod \"opendatahub-operator-controller-manager-58d8b99777-qwkg6\" (UID: \"c379fead-1cd2-4d3c-8327-e96957a5fb6b\") " pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.571250 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.571211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:13:54.667571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.667505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.762835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.762795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.852029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.852001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6"] Sep 16 14:13:54.853673 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.853645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.855475 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:13:54.855449 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc379fead_1cd2_4d3c_8327_e96957a5fb6b.slice/crio-b7ae05d953ee2586c8207e0b2124bf51ee58d3ebcb8d4e4213f2d368e557f5ac WatchSource:0}: Error finding container b7ae05d953ee2586c8207e0b2124bf51ee58d3ebcb8d4e4213f2d368e557f5ac: Status 404 returned error can't find the container with id b7ae05d953ee2586c8207e0b2124bf51ee58d3ebcb8d4e4213f2d368e557f5ac Sep 16 14:13:54.944294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:54.944246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:55.040230 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:55.040201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:55.417650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:55.417556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" event={"ID":"c379fead-1cd2-4d3c-8327-e96957a5fb6b","Type":"ContainerStarted","Data":"b7ae05d953ee2586c8207e0b2124bf51ee58d3ebcb8d4e4213f2d368e557f5ac"} Sep 16 14:13:58.435937 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:58.435907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" event={"ID":"c379fead-1cd2-4d3c-8327-e96957a5fb6b","Type":"ContainerStarted","Data":"ab8af9300c1a14a59259ce736950b6319e805d687253b3d3d33c8a9de0e1a635"} Sep 16 14:13:59.442063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:59.442028 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerID="8bbfbacc5a73fcc18469d36a1da52617151e1d2bb5a370df3adbb3134597d6fc" exitCode=0 Sep 16 14:13:59.442515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:59.442109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" event={"ID":"5e3a17df-ae39-4e28-b395-31d96165ca45","Type":"ContainerDied","Data":"8bbfbacc5a73fcc18469d36a1da52617151e1d2bb5a370df3adbb3134597d6fc"} Sep 16 14:13:59.443889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:59.443859 2579 generic.go:356] "Generic (PLEG): container finished" podID="c379fead-1cd2-4d3c-8327-e96957a5fb6b" containerID="ab8af9300c1a14a59259ce736950b6319e805d687253b3d3d33c8a9de0e1a635" exitCode=0 Sep 16 14:13:59.443991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:13:59.443896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" event={"ID":"c379fead-1cd2-4d3c-8327-e96957a5fb6b","Type":"ContainerDied","Data":"ab8af9300c1a14a59259ce736950b6319e805d687253b3d3d33c8a9de0e1a635"} Sep 16 14:14:00.451203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:00.451164 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerID="ea8b545b7d6091db2da0a273f35bcfd4d7477beb2cea6d84d00905114fb80503" exitCode=0 Sep 16 14:14:00.451681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:00.451238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" event={"ID":"5e3a17df-ae39-4e28-b395-31d96165ca45","Type":"ContainerDied","Data":"ea8b545b7d6091db2da0a273f35bcfd4d7477beb2cea6d84d00905114fb80503"} Sep 16 14:14:01.458140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.458098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" event={"ID":"c379fead-1cd2-4d3c-8327-e96957a5fb6b","Type":"ContainerStarted","Data":"88e94aaccb3689b743d3d5406fc98d7f7fd366a9918b2226b9569b5fd790d490"} Sep 16 14:14:01.458694 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.458249 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:14:01.493646 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.493574 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" podStartSLOduration=2.127086494 podStartE2EDuration="7.493557087s" totalImagesPullingTime="4.325639673s" totalInitContainerRuntime="1.04083092s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:01.491765555 +0000 UTC m=+372.069126707" watchObservedRunningTime="2026-09-16 14:14:01.493557087 +0000 UTC m=+372.070918239" Sep 16 14:14:01.585931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.585894 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:14:01.662005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.661972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle\") pod \"5e3a17df-ae39-4e28-b395-31d96165ca45\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " Sep 16 14:14:01.662174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.662067 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5hgz\" (UniqueName: \"kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz\") pod \"5e3a17df-ae39-4e28-b395-31d96165ca45\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " Sep 16 14:14:01.662174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.662111 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util\") pod \"5e3a17df-ae39-4e28-b395-31d96165ca45\" (UID: \"5e3a17df-ae39-4e28-b395-31d96165ca45\") " Sep 16 14:14:01.662781 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.662746 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle" (OuterVolumeSpecName: "bundle") pod "5e3a17df-ae39-4e28-b395-31d96165ca45" (UID: "5e3a17df-ae39-4e28-b395-31d96165ca45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:01.664291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.664270 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz" (OuterVolumeSpecName: "kube-api-access-w5hgz") pod "5e3a17df-ae39-4e28-b395-31d96165ca45" (UID: "5e3a17df-ae39-4e28-b395-31d96165ca45"). InnerVolumeSpecName "kube-api-access-w5hgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:01.668221 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.668196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util" (OuterVolumeSpecName: "util") pod "5e3a17df-ae39-4e28-b395-31d96165ca45" (UID: "5e3a17df-ae39-4e28-b395-31d96165ca45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:01.763662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.763582 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:01.763662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.763629 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3a17df-ae39-4e28-b395-31d96165ca45-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:01.763662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:01.763649 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hgz\" (UniqueName: \"kubernetes.io/projected/5e3a17df-ae39-4e28-b395-31d96165ca45-kube-api-access-w5hgz\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:02.464404 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:02.464372 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" Sep 16 14:14:02.464884 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:02.464404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0f2ppw" event={"ID":"5e3a17df-ae39-4e28-b395-31d96165ca45","Type":"ContainerDied","Data":"3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d"} Sep 16 14:14:02.464884 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:02.464448 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed15d951c5f6cef8acffe3c80377a69049c62cad81f39606c3374bde17ced3d" Sep 16 14:14:06.157360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:06.157324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:06.250487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:06.250451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:08.427048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff9f4c48b-n4n62"] Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427344 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="util" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427356 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="util" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427367 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="pull" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427372 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="pull" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427383 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="extract" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427391 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="extract" Sep 16 14:14:08.427518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.427456 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3a17df-ae39-4e28-b395-31d96165ca45" containerName="extract" Sep 16 14:14:08.435409 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.435387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.445161 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.445129 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9f4c48b-n4n62"] Sep 16 14:14:08.527186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-trusted-ca-bundle\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-oauth-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcn8c\" (UniqueName: \"kubernetes.io/projected/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-kube-api-access-bcn8c\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-oauth-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.527478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.527416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-service-ca\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.531824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.531797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:08.628101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-trusted-ca-bundle\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-oauth-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcn8c\" (UniqueName: \"kubernetes.io/projected/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-kube-api-access-bcn8c\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-oauth-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.628518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.628264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-service-ca\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.629096 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.629068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-oauth-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.629223 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.629171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-trusted-ca-bundle\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.629310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.629287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-service-ca\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.629350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.629294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.630980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.630951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-oauth-config\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.631076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.631013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-console-serving-cert\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.634334 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.634287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:08.639697 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.639676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcn8c\" (UniqueName: \"kubernetes.io/projected/ab1abec4-ace6-4c90-b657-0d87d59aeb3c-kube-api-access-bcn8c\") pod \"console-5ff9f4c48b-n4n62\" (UID: \"ab1abec4-ace6-4c90-b657-0d87d59aeb3c\") " pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:08.745747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:08.745669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:09.034041 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:09.034011 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9f4c48b-n4n62"] Sep 16 14:14:09.036514 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:09.036491 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1abec4_ace6_4c90_b657_0d87d59aeb3c.slice/crio-5281cd816f3851fc5d1735171dc38394207845444e4b93eb7bbfffdb81d21af4 WatchSource:0}: Error finding container 5281cd816f3851fc5d1735171dc38394207845444e4b93eb7bbfffdb81d21af4: Status 404 returned error can't find the container with id 5281cd816f3851fc5d1735171dc38394207845444e4b93eb7bbfffdb81d21af4 Sep 16 14:14:09.499616 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:09.499577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9f4c48b-n4n62" event={"ID":"ab1abec4-ace6-4c90-b657-0d87d59aeb3c","Type":"ContainerStarted","Data":"41ce051bd986e4ed13a219e11933d4d215a0fe3d557e9ca1241df9113d0e004a"} Sep 16 14:14:09.500006 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:09.499645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9f4c48b-n4n62" event={"ID":"ab1abec4-ace6-4c90-b657-0d87d59aeb3c","Type":"ContainerStarted","Data":"5281cd816f3851fc5d1735171dc38394207845444e4b93eb7bbfffdb81d21af4"} Sep 16 14:14:09.524784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:09.524736 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff9f4c48b-n4n62" podStartSLOduration=1.524719183 podStartE2EDuration="1.524719183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:09.523492862 +0000 UTC m=+380.100854013" watchObservedRunningTime="2026-09-16 14:14:09.524719183 +0000 UTC m=+380.102080335" Sep 16 14:14:12.467250 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:12.467211 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58d8b99777-qwkg6" Sep 16 14:14:18.271857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.271810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:18.390990 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.390958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:18.666633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.666529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:18.745982 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.745943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:18.746120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.746011 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:18.751466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.751443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:18.763826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.763801 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265"] Sep 16 14:14:18.767172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.767154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:18.770149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.770128 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 14:14:18.770246 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.770128 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 14:14:18.770246 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.770228 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hbc8q\"" Sep 16 14:14:18.785123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.785098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265"] Sep 16 14:14:18.898504 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.898464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:18.915500 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.915471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkbp\" (UniqueName: \"kubernetes.io/projected/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-kube-api-access-ggkbp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:18.915676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.915514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tls-certs\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:18.915676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:18.915579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tmp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.016469 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.016431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkbp\" (UniqueName: \"kubernetes.io/projected/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-kube-api-access-ggkbp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.016666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.016497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tls-certs\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.016666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.016532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tmp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.018945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.018924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tmp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.019121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.019102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-tls-certs\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.033908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.033882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkbp\" (UniqueName: \"kubernetes.io/projected/869a4d91-529e-461c-aa3a-fec0ec4c0a0c-kube-api-access-ggkbp\") pod \"kube-auth-proxy-57bfccdb6c-pb265\" (UID: \"869a4d91-529e-461c-aa3a-fec0ec4c0a0c\") " pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.077742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.077719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" Sep 16 14:14:19.361994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.361967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265"] Sep 16 14:14:19.364699 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:19.364670 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869a4d91_529e_461c_aa3a_fec0ec4c0a0c.slice/crio-27d3d189d268acadf0bad804ceb348921e761e6aae1e238930ff7d322dbfb23c WatchSource:0}: Error finding container 27d3d189d268acadf0bad804ceb348921e761e6aae1e238930ff7d322dbfb23c: Status 404 returned error can't find the container with id 27d3d189d268acadf0bad804ceb348921e761e6aae1e238930ff7d322dbfb23c Sep 16 14:14:19.546994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.546957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" event={"ID":"869a4d91-529e-461c-aa3a-fec0ec4c0a0c","Type":"ContainerStarted","Data":"27d3d189d268acadf0bad804ceb348921e761e6aae1e238930ff7d322dbfb23c"} Sep 16 14:14:19.551088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.551055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff9f4c48b-n4n62" Sep 16 14:14:19.661487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.661409 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:14:19.881596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.881566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:19.984196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:19.984162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:20.421836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.421704 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:20.426035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.426012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.429821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.428938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 14:14:20.429821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.429156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 14:14:20.430534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.430036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 14:14:20.430534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.430374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bbsvf\"" Sep 16 14:14:20.431537 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.431085 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 14:14:20.431537 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.431479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:20.445755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.445729 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:20.529256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.529206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:20.531273 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2lm\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.531897 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.531516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.632631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.632820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.632820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.632820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.632820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.633028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.633028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.632858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2lm\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.633429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.633389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.635680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.635651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.635894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.635874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.636194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.636161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.636509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.636488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.642005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.641986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.642559 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.642538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2lm\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm\") pod \"istiod-openshift-gateway-7b7995d5cd-jtfkk\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:20.740524 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:20.740493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:21.040076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:21.040050 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:22.035088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:22.035045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:22.139747 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:22.139716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaf4bb7_e63e_4112_84a0_b5c0e66797db.slice/crio-7d7105b44fa5a5cc78a0998e0e6c550b81ca318ff1b8cfc92b429c949b1b536b WatchSource:0}: Error finding container 7d7105b44fa5a5cc78a0998e0e6c550b81ca318ff1b8cfc92b429c949b1b536b: Status 404 returned error can't find the container with id 7d7105b44fa5a5cc78a0998e0e6c550b81ca318ff1b8cfc92b429c949b1b536b Sep 16 14:14:22.566263 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:22.566228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" event={"ID":"869a4d91-529e-461c-aa3a-fec0ec4c0a0c","Type":"ContainerStarted","Data":"58eb7e8111e6c8feb8f12329dc77e8e58db46c01cf82ac07a46a42d173c7b4ed"} Sep 16 14:14:22.567635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:22.567592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" event={"ID":"7aaf4bb7-e63e-4112-84a0-b5c0e66797db","Type":"ContainerStarted","Data":"7d7105b44fa5a5cc78a0998e0e6c550b81ca318ff1b8cfc92b429c949b1b536b"} Sep 16 14:14:22.587972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:22.587920 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57bfccdb6c-pb265" podStartSLOduration=1.7602960909999998 podStartE2EDuration="4.587900378s" totalImagesPullingTime="2.827604287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:22.586707415 +0000 UTC m=+393.164068559" watchObservedRunningTime="2026-09-16 14:14:22.587900378 +0000 UTC m=+393.165261530" Sep 16 14:14:25.339983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:25.339945 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:14:25.340282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:25.340012 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:14:25.585344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:25.585298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" event={"ID":"7aaf4bb7-e63e-4112-84a0-b5c0e66797db","Type":"ContainerStarted","Data":"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11"} Sep 16 14:14:25.585527 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:25.585493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:25.618834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:25.618736 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" podStartSLOduration=2.420747128 podStartE2EDuration="5.618715993s" totalImagesPullingTime="3.197968865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:25.616522438 +0000 UTC m=+396.193883601" watchObservedRunningTime="2026-09-16 14:14:25.618715993 +0000 UTC m=+396.196077151" Sep 16 14:14:26.111897 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:26.111856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:26.217148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:26.217105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:26.592098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:26.592065 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-jtfkk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 14:14:26.592481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:26.592119 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 14:14:27.010318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:27.010279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:27.230846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:27.230813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:27.328304 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:27.328220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:29.253490 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.253451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:29.343783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.343752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:29.591674 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.591583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:29.817539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.817512 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tdqcn"] Sep 16 14:14:29.822003 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.821987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:29.824697 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.824676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 16 14:14:29.838221 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.838197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tdqcn"] Sep 16 14:14:29.848290 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.848221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:29.911855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.911813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqx2\" (UniqueName: \"kubernetes.io/projected/99e6de89-cdab-47c6-82ae-afece8adb6ea-kube-api-access-scqx2\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:29.912043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.911891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e6de89-cdab-47c6-82ae-afece8adb6ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:29.940194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:29.940163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:30.012580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.012545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqx2\" (UniqueName: \"kubernetes.io/projected/99e6de89-cdab-47c6-82ae-afece8adb6ea-kube-api-access-scqx2\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:30.012795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.012594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e6de89-cdab-47c6-82ae-afece8adb6ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:30.015180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.015157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e6de89-cdab-47c6-82ae-afece8adb6ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:30.022916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.022891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqx2\" (UniqueName: \"kubernetes.io/projected/99e6de89-cdab-47c6-82ae-afece8adb6ea-kube-api-access-scqx2\") pod \"model-serving-api-54d89bd5fc-tdqcn\" (UID: \"99e6de89-cdab-47c6-82ae-afece8adb6ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:30.134170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.134095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:30.431939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.431906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tdqcn"] Sep 16 14:14:30.434321 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:30.434285 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e6de89_cdab_47c6_82ae_afece8adb6ea.slice/crio-9a9edc0c72d14c8ec1e26a634ba32207bb06481ae9280b65363c874f189706a6 WatchSource:0}: Error finding container 9a9edc0c72d14c8ec1e26a634ba32207bb06481ae9280b65363c874f189706a6: Status 404 returned error can't find the container with id 9a9edc0c72d14c8ec1e26a634ba32207bb06481ae9280b65363c874f189706a6 Sep 16 14:14:30.610296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:30.610268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" event={"ID":"99e6de89-cdab-47c6-82ae-afece8adb6ea","Type":"ContainerStarted","Data":"9a9edc0c72d14c8ec1e26a634ba32207bb06481ae9280b65363c874f189706a6"} Sep 16 14:14:32.392791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:32.392761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:32.620919 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:32.620836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" event={"ID":"99e6de89-cdab-47c6-82ae-afece8adb6ea","Type":"ContainerStarted","Data":"ca6b0d0ba55fee0c8cc5407c4fbcc39da6cb9fd66ec896b12f98e6ae79f8b811"} Sep 16 14:14:32.620919 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:32.620896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:32.648513 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:32.648464 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" podStartSLOduration=1.811930062 podStartE2EDuration="3.64844874s" totalImagesPullingTime="1.836518678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:32.64799034 +0000 UTC m=+403.225351495" watchObservedRunningTime="2026-09-16 14:14:32.64844874 +0000 UTC m=+403.225809892" Sep 16 14:14:39.986893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:39.986850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:40.056255 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.056223 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6"] Sep 16 14:14:40.059764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.059745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.065483 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.065461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fphsp\"" Sep 16 14:14:40.074834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.074814 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 16 14:14:40.083263 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.083242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.083349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.083303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c669c\" (UniqueName: \"kubernetes.io/projected/d6f0b322-e108-4804-bef5-09664f5b22eb-kube-api-access-c669c\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.086630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.086593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6"] Sep 16 14:14:40.125225 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.125196 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r"] Sep 16 14:14:40.128573 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.128554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.132536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.132515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 16 14:14:40.132665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.132620 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rg99v\"" Sep 16 14:14:40.159328 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.159303 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r"] Sep 16 14:14:40.184376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.184350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.184506 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.184391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplk7\" (UniqueName: \"kubernetes.io/projected/9bdbd6c4-57d5-4946-9e16-802ac7536446-kube-api-access-jplk7\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.184506 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:14:40.184477 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 16 14:14:40.184600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.184503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c669c\" (UniqueName: \"kubernetes.io/projected/d6f0b322-e108-4804-bef5-09664f5b22eb-kube-api-access-c669c\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.184600 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:14:40.184529 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert podName:d6f0b322-e108-4804-bef5-09664f5b22eb nodeName:}" failed. No retries permitted until 2026-09-16 14:14:40.684513199 +0000 UTC m=+411.261874332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert") pod "kserve-controller-manager-d6f4b65b8-7z5t6" (UID: "d6f0b322-e108-4804-bef5-09664f5b22eb") : secret "kserve-webhook-server-cert" not found Sep 16 14:14:40.184600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.184552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bdbd6c4-57d5-4946-9e16-802ac7536446-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.228541 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.228509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c669c\" (UniqueName: \"kubernetes.io/projected/d6f0b322-e108-4804-bef5-09664f5b22eb-kube-api-access-c669c\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.285597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.285525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk7\" (UniqueName: \"kubernetes.io/projected/9bdbd6c4-57d5-4946-9e16-802ac7536446-kube-api-access-jplk7\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.285750 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.285633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bdbd6c4-57d5-4946-9e16-802ac7536446-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.288032 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.288010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bdbd6c4-57d5-4946-9e16-802ac7536446-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.300218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.300195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplk7\" (UniqueName: \"kubernetes.io/projected/9bdbd6c4-57d5-4946-9e16-802ac7536446-kube-api-access-jplk7\") pod \"llmisvc-controller-manager-777c6bd7d5-2288r\" (UID: \"9bdbd6c4-57d5-4946-9e16-802ac7536446\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.381904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.381877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:40.438690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.438669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:40.540295 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.540211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:40.688386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.688350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.691339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.691313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f0b322-e108-4804-bef5-09664f5b22eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-7z5t6\" (UID: \"d6f0b322-e108-4804-bef5-09664f5b22eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:40.696041 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.696007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:40.715187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.715167 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r"] Sep 16 14:14:40.716804 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:40.716776 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9bdbd6c4_57d5_4946_9e16_802ac7536446.slice/crio-ee916f8f5f60d42393c4c02a5ffe18331b25667805315a554ebb90eae4712de9 WatchSource:0}: Error finding container ee916f8f5f60d42393c4c02a5ffe18331b25667805315a554ebb90eae4712de9: Status 404 returned error can't find the container with id ee916f8f5f60d42393c4c02a5ffe18331b25667805315a554ebb90eae4712de9 Sep 16 14:14:40.950802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.950770 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:40.950962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.950942 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerName="discovery" containerID="cri-o://174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11" gracePeriod=30 Sep 16 14:14:40.970817 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:40.970791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:41.185254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.185230 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:41.191405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191424 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191446 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f2lm\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.191732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191651 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig\") pod \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\" (UID: \"7aaf4bb7-e63e-4112-84a0-b5c0e66797db\") " Sep 16 14:14:41.192414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:41.192414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.191912 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-ca-configmap\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.193885 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.193832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts" (OuterVolumeSpecName: "cacerts") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:41.193994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.193939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs" (OuterVolumeSpecName: "local-certs") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:41.194268 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.194243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm" (OuterVolumeSpecName: "kube-api-access-2f2lm") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "kube-api-access-2f2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:41.194268 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.194262 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token" (OuterVolumeSpecName: "istio-token") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:41.194565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.194276 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:41.194565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.194330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7aaf4bb7-e63e-4112-84a0-b5c0e66797db" (UID: "7aaf4bb7-e63e-4112-84a0-b5c0e66797db"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:41.247536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.247507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6"] Sep 16 14:14:41.249441 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:14:41.249412 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f0b322_e108_4804_bef5_09664f5b22eb.slice/crio-5006d32834968dd49cbb1d7b1dde6b586c6d2afb44e973be5e16e36476e938d8 WatchSource:0}: Error finding container 5006d32834968dd49cbb1d7b1dde6b586c6d2afb44e973be5e16e36476e938d8: Status 404 returned error can't find the container with id 5006d32834968dd49cbb1d7b1dde6b586c6d2afb44e973be5e16e36476e938d8 Sep 16 14:14:41.292664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292629 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-csr-dns-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.292664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292656 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-local-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.292664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292665 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-kubeconfig\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.292883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292675 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-istio-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.292883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292689 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2lm\" (UniqueName: \"kubernetes.io/projected/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-kube-api-access-2f2lm\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.292883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.292700 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7aaf4bb7-e63e-4112-84a0-b5c0e66797db-cacerts\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:41.354943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.354906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:41.668881 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.668786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" event={"ID":"d6f0b322-e108-4804-bef5-09664f5b22eb","Type":"ContainerStarted","Data":"5006d32834968dd49cbb1d7b1dde6b586c6d2afb44e973be5e16e36476e938d8"} Sep 16 14:14:41.670278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.670249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" event={"ID":"9bdbd6c4-57d5-4946-9e16-802ac7536446","Type":"ContainerStarted","Data":"ee916f8f5f60d42393c4c02a5ffe18331b25667805315a554ebb90eae4712de9"} Sep 16 14:14:41.671575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.671546 2579 generic.go:356] "Generic (PLEG): container finished" podID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerID="174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11" exitCode=0 Sep 16 14:14:41.671717 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.671617 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" Sep 16 14:14:41.671794 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.671627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" event={"ID":"7aaf4bb7-e63e-4112-84a0-b5c0e66797db","Type":"ContainerDied","Data":"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11"} Sep 16 14:14:41.671794 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.671756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk" event={"ID":"7aaf4bb7-e63e-4112-84a0-b5c0e66797db","Type":"ContainerDied","Data":"7d7105b44fa5a5cc78a0998e0e6c550b81ca318ff1b8cfc92b429c949b1b536b"} Sep 16 14:14:41.671794 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.671788 2579 scope.go:117] "RemoveContainer" containerID="174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11" Sep 16 14:14:41.718071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.718038 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:41.725348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.725320 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jtfkk"] Sep 16 14:14:41.754502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.754478 2579 scope.go:117] "RemoveContainer" containerID="174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11" Sep 16 14:14:41.754903 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:14:41.754881 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11\": container with ID starting with 174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11 not found: ID does not exist" containerID="174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11" Sep 16 14:14:41.754976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.754914 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11"} err="failed to get container status \"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11\": rpc error: code = NotFound desc = could not find container \"174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11\": container with ID starting with 174059a7a634b79a08d8ae20bd47dfba5d26359e6a3280f4dafd81536a54af11 not found: ID does not exist" Sep 16 14:14:41.825223 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.821441 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" path="/var/lib/kubelet/pods/7aaf4bb7-e63e-4112-84a0-b5c0e66797db/volumes" Sep 16 14:14:41.895853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.895795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:41.999103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:41.999056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:43.630332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:43.630304 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tdqcn" Sep 16 14:14:44.682055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.682013 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bb7f7fb9-v26gc" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" containerName="console" containerID="cri-o://4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5" gracePeriod=15 Sep 16 14:14:44.691491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.691458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" event={"ID":"d6f0b322-e108-4804-bef5-09664f5b22eb","Type":"ContainerStarted","Data":"cee6e6722f3126a8ffb9812f0463916ab17109ac8395980c5940f9947d94910a"} Sep 16 14:14:44.691661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.691527 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:14:44.693157 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.693123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" event={"ID":"9bdbd6c4-57d5-4946-9e16-802ac7536446","Type":"ContainerStarted","Data":"2b4e4980d2972966229d262649389e997fb990421ef3603ef57c257fba462d5e"} Sep 16 14:14:44.693302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.693287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:14:44.720444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.720403 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" podStartSLOduration=1.835618046 podStartE2EDuration="4.720387311s" totalImagesPullingTime="2.884769265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:44.718230964 +0000 UTC m=+415.295592113" watchObservedRunningTime="2026-09-16 14:14:44.720387311 +0000 UTC m=+415.297748463" Sep 16 14:14:44.930388 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.930366 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:14:44.954654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:44.954538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" podStartSLOduration=1.538798523 podStartE2EDuration="4.954521628s" totalImagesPullingTime="3.415723105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:44.753567411 +0000 UTC m=+415.330928563" watchObservedRunningTime="2026-09-16 14:14:44.954521628 +0000 UTC m=+415.531882781" Sep 16 14:14:45.023493 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023569 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023598 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt4lp\" (UniqueName: \"kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023645 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.023853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert\") pod \"f809fdf0-46a6-45ee-975f-eaabed778bec\" (UID: \"f809fdf0-46a6-45ee-975f-eaabed778bec\") " Sep 16 14:14:45.024165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.023954 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca" (OuterVolumeSpecName: "service-ca") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:45.024165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.024110 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:45.024165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.024130 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config" (OuterVolumeSpecName: "console-config") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:45.024165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.024142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:45.026036 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.026010 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:45.026145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.026065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:45.026251 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.026233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp" (OuterVolumeSpecName: "kube-api-access-vt4lp") pod "f809fdf0-46a6-45ee-975f-eaabed778bec" (UID: "f809fdf0-46a6-45ee-975f-eaabed778bec"). InnerVolumeSpecName "kube-api-access-vt4lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:45.125123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125086 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-trusted-ca-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125115 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4lp\" (UniqueName: \"kubernetes.io/projected/f809fdf0-46a6-45ee-975f-eaabed778bec-kube-api-access-vt4lp\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125124 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-console-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125133 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125141 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-oauth-serving-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125150 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f809fdf0-46a6-45ee-975f-eaabed778bec-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.125461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.125157 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f809fdf0-46a6-45ee-975f-eaabed778bec-console-oauth-config\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:14:45.698893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.698858 2579 generic.go:356] "Generic (PLEG): container finished" podID="f809fdf0-46a6-45ee-975f-eaabed778bec" containerID="4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5" exitCode=0 Sep 16 14:14:45.699338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.698900 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb7f7fb9-v26gc" Sep 16 14:14:45.699338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.698942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb7f7fb9-v26gc" event={"ID":"f809fdf0-46a6-45ee-975f-eaabed778bec","Type":"ContainerDied","Data":"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5"} Sep 16 14:14:45.699338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.698990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb7f7fb9-v26gc" event={"ID":"f809fdf0-46a6-45ee-975f-eaabed778bec","Type":"ContainerDied","Data":"686313e6b07e52bcdda46f9fba8df4c1c51aef63f01d836e467a0cfb44a97a00"} Sep 16 14:14:45.699338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.699009 2579 scope.go:117] "RemoveContainer" containerID="4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5" Sep 16 14:14:45.707965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.707948 2579 scope.go:117] "RemoveContainer" containerID="4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5" Sep 16 14:14:45.708228 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:14:45.708210 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5\": container with ID starting with 4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5 not found: ID does not exist" containerID="4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5" Sep 16 14:14:45.708267 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.708236 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5"} err="failed to get container status \"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5\": rpc error: code = NotFound desc = could not find container \"4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5\": container with ID starting with 4c64eb8e64c6a8a920014fb240761abef05028cb63906f20abb21114e07996a5 not found: ID does not exist" Sep 16 14:14:45.756423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.756389 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:14:45.771207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.771182 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb7f7fb9-v26gc"] Sep 16 14:14:45.817142 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:14:45.817112 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" path="/var/lib/kubelet/pods/f809fdf0-46a6-45ee-975f-eaabed778bec/volumes" Sep 16 14:15:15.701825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:15.701745 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2288r" Sep 16 14:15:15.704675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:15.704657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7z5t6" Sep 16 14:15:30.409132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:30.409095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:30.492837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:30.492808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:35.129832 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.129773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw"] Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130133 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerName="discovery" Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130145 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerName="discovery" Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130164 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" containerName="console" Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" containerName="console" Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130225 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f809fdf0-46a6-45ee-975f-eaabed778bec" containerName="console" Sep 16 14:15:35.130287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.130235 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aaf4bb7-e63e-4112-84a0-b5c0e66797db" containerName="discovery" Sep 16 14:15:35.136946 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.136926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.140097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.140074 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 14:15:35.140236 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.140120 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 14:15:35.141291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.141264 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5wbqx\"" Sep 16 14:15:35.144872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.144853 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw"] Sep 16 14:15:35.224752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.224721 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g"] Sep 16 14:15:35.228303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.228288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.236427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.236404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g"] Sep 16 14:15:35.240702 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.240681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtkx\" (UniqueName: \"kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.240800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.240719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.240800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.240737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.324397 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.324364 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9"] Sep 16 14:15:35.328210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.328193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.338313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.338284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9"] Sep 16 14:15:35.341555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtkx\" (UniqueName: \"kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.341693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.341693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.341693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.341693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.341909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.341695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4gl\" (UniqueName: \"kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.342025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.342006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.342095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.342070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.352730 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.352709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtkx\" (UniqueName: \"kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.423060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.423036 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5"] Sep 16 14:15:35.426700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.426683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.442223 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.442344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.442344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.442344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.442344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4gl\" (UniqueName: \"kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.442585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbh7\" (UniqueName: \"kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.442677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.442677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.442629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.443626 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.443587 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5"] Sep 16 14:15:35.447154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.447132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:35.456123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.456096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4gl\" (UniqueName: \"kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.538688 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.538656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:35.543024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.542828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdsx\" (UniqueName: \"kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.543024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.542857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.543024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.542992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.543209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.543043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.543209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.543086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.543209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.543122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbh7\" (UniqueName: \"kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.543393 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.543371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.543454 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.543381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.555523 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.555498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbh7\" (UniqueName: \"kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.637729 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.637691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:35.644175 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.644146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.644314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.644243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdsx\" (UniqueName: \"kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.644314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.644277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.644510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.644489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.644624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.644581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.658156 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.658133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdsx\" (UniqueName: \"kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.729786 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.729761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw"] Sep 16 14:15:35.732161 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:35.732125 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf043c50a_d9a4_4310_a3a3_794afee677b2.slice/crio-9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f WatchSource:0}: Error finding container 9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f: Status 404 returned error can't find the container with id 9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f Sep 16 14:15:35.735383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.735364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:35.824748 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.824716 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g"] Sep 16 14:15:35.828342 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:35.828303 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1372bfb4_3967_4030_84cb_5f6cfd5bb3a1.slice/crio-fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a WatchSource:0}: Error finding container fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a: Status 404 returned error can't find the container with id fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a Sep 16 14:15:35.920997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.920975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9"] Sep 16 14:15:35.931071 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:35.931048 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d8b426_a116_44db_ae14_3242e17611b4.slice/crio-913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338 WatchSource:0}: Error finding container 913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338: Status 404 returned error can't find the container with id 913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338 Sep 16 14:15:35.952968 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.952941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" event={"ID":"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1","Type":"ContainerStarted","Data":"fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a"} Sep 16 14:15:35.954244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.954221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" event={"ID":"15d8b426-a116-44db-ae14-3242e17611b4","Type":"ContainerStarted","Data":"913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338"} Sep 16 14:15:35.955924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.955903 2579 generic.go:356] "Generic (PLEG): container finished" podID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerID="4e441975fda6ad5ea27b34a9be0081c548dc0d8dd8a9e3b594695f30a44621f1" exitCode=0 Sep 16 14:15:35.956033 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.955990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" event={"ID":"f043c50a-d9a4-4310-a3a3-794afee677b2","Type":"ContainerDied","Data":"4e441975fda6ad5ea27b34a9be0081c548dc0d8dd8a9e3b594695f30a44621f1"} Sep 16 14:15:35.956090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:35.956039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" event={"ID":"f043c50a-d9a4-4310-a3a3-794afee677b2","Type":"ContainerStarted","Data":"9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f"} Sep 16 14:15:36.014893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.014872 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5"] Sep 16 14:15:36.098383 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:36.098355 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9745b4_e14c_4cb8_aefb_a794375f28cf.slice/crio-161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390 WatchSource:0}: Error finding container 161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390: Status 404 returned error can't find the container with id 161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390 Sep 16 14:15:36.962432 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.962393 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerID="cf5b4de6bc7e9867f1ac55d9d5d098dbe78286ccaacfd4986b7a5dd1cecb9ca1" exitCode=0 Sep 16 14:15:36.962854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.962485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" event={"ID":"1d9745b4-e14c-4cb8-aefb-a794375f28cf","Type":"ContainerDied","Data":"cf5b4de6bc7e9867f1ac55d9d5d098dbe78286ccaacfd4986b7a5dd1cecb9ca1"} Sep 16 14:15:36.962854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.962538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" event={"ID":"1d9745b4-e14c-4cb8-aefb-a794375f28cf","Type":"ContainerStarted","Data":"161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390"} Sep 16 14:15:36.964306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.964288 2579 generic.go:356] "Generic (PLEG): container finished" podID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerID="616a63d77825881cd87b745a4d439a3dc03fa2ac2fc4013d0bed1ad4039dce17" exitCode=0 Sep 16 14:15:36.964384 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.964316 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" event={"ID":"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1","Type":"ContainerDied","Data":"616a63d77825881cd87b745a4d439a3dc03fa2ac2fc4013d0bed1ad4039dce17"} Sep 16 14:15:36.966220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.966121 2579 generic.go:356] "Generic (PLEG): container finished" podID="15d8b426-a116-44db-ae14-3242e17611b4" containerID="56df8d1f706bb6165e09f4d01761b89b2770a72572c315d601ed6a36af7e51fd" exitCode=0 Sep 16 14:15:36.966220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.966205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" event={"ID":"15d8b426-a116-44db-ae14-3242e17611b4","Type":"ContainerDied","Data":"56df8d1f706bb6165e09f4d01761b89b2770a72572c315d601ed6a36af7e51fd"} Sep 16 14:15:36.968343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.968325 2579 generic.go:356] "Generic (PLEG): container finished" podID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerID="59fbe289aeb112b84ad15055fb38b7bb032a59bae20a0b347ec8169d7e6ba224" exitCode=0 Sep 16 14:15:36.968427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:36.968390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" event={"ID":"f043c50a-d9a4-4310-a3a3-794afee677b2","Type":"ContainerDied","Data":"59fbe289aeb112b84ad15055fb38b7bb032a59bae20a0b347ec8169d7e6ba224"} Sep 16 14:15:37.977776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:37.977738 2579 generic.go:356] "Generic (PLEG): container finished" podID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerID="b72c7703df37b8a7625f57b8ba2b8447d7d2f696a3d236e2a31e8b83798b55ac" exitCode=0 Sep 16 14:15:37.978108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:37.977803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" event={"ID":"f043c50a-d9a4-4310-a3a3-794afee677b2","Type":"ContainerDied","Data":"b72c7703df37b8a7625f57b8ba2b8447d7d2f696a3d236e2a31e8b83798b55ac"} Sep 16 14:15:38.984986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.984942 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerID="07229c7c679ca900faa102f6d6d2ae891691dbf6ebe6c6c1d086b2528f5d197a" exitCode=0 Sep 16 14:15:38.985386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.985033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" event={"ID":"1d9745b4-e14c-4cb8-aefb-a794375f28cf","Type":"ContainerDied","Data":"07229c7c679ca900faa102f6d6d2ae891691dbf6ebe6c6c1d086b2528f5d197a"} Sep 16 14:15:38.987052 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.987029 2579 generic.go:356] "Generic (PLEG): container finished" podID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerID="4dc9521785590d42b1d7b13751630071c01ab2c5ea46068fa624992d01e21b9d" exitCode=0 Sep 16 14:15:38.987155 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.987106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" event={"ID":"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1","Type":"ContainerDied","Data":"4dc9521785590d42b1d7b13751630071c01ab2c5ea46068fa624992d01e21b9d"} Sep 16 14:15:38.989094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.989073 2579 generic.go:356] "Generic (PLEG): container finished" podID="15d8b426-a116-44db-ae14-3242e17611b4" containerID="daedd03f5b40a0541970e82c6806b7cc83b92fd300fa51d703307e3252cb2775" exitCode=0 Sep 16 14:15:38.989180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:38.989156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" event={"ID":"15d8b426-a116-44db-ae14-3242e17611b4","Type":"ContainerDied","Data":"daedd03f5b40a0541970e82c6806b7cc83b92fd300fa51d703307e3252cb2775"} Sep 16 14:15:39.141159 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.141064 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:39.276598 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.276568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vtkx\" (UniqueName: \"kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx\") pod \"f043c50a-d9a4-4310-a3a3-794afee677b2\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " Sep 16 14:15:39.276785 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.276659 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle\") pod \"f043c50a-d9a4-4310-a3a3-794afee677b2\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " Sep 16 14:15:39.276785 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.276703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util\") pod \"f043c50a-d9a4-4310-a3a3-794afee677b2\" (UID: \"f043c50a-d9a4-4310-a3a3-794afee677b2\") " Sep 16 14:15:39.277303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.277276 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle" (OuterVolumeSpecName: "bundle") pod "f043c50a-d9a4-4310-a3a3-794afee677b2" (UID: "f043c50a-d9a4-4310-a3a3-794afee677b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:39.278769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.278744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx" (OuterVolumeSpecName: "kube-api-access-5vtkx") pod "f043c50a-d9a4-4310-a3a3-794afee677b2" (UID: "f043c50a-d9a4-4310-a3a3-794afee677b2"). InnerVolumeSpecName "kube-api-access-5vtkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:39.281755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.281731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util" (OuterVolumeSpecName: "util") pod "f043c50a-d9a4-4310-a3a3-794afee677b2" (UID: "f043c50a-d9a4-4310-a3a3-794afee677b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:39.377287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.377253 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:39.377287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.377279 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f043c50a-d9a4-4310-a3a3-794afee677b2-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:39.377287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.377290 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtkx\" (UniqueName: \"kubernetes.io/projected/f043c50a-d9a4-4310-a3a3-794afee677b2-kube-api-access-5vtkx\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:39.996472 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.996441 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerID="3b3f47606d56cb7bbf959eed6ea52191dc9f4a77a69818b2cf818ea67548b0ce" exitCode=0 Sep 16 14:15:39.996868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.996520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" event={"ID":"1d9745b4-e14c-4cb8-aefb-a794375f28cf","Type":"ContainerDied","Data":"3b3f47606d56cb7bbf959eed6ea52191dc9f4a77a69818b2cf818ea67548b0ce"} Sep 16 14:15:39.998711 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.998688 2579 generic.go:356] "Generic (PLEG): container finished" podID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerID="2b83ed2380591a7ec2db392a6dd8fc4607b4572396e9e9f21d448556693ccc47" exitCode=0 Sep 16 14:15:39.998808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:39.998773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" event={"ID":"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1","Type":"ContainerDied","Data":"2b83ed2380591a7ec2db392a6dd8fc4607b4572396e9e9f21d448556693ccc47"} Sep 16 14:15:40.000915 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:40.000893 2579 generic.go:356] "Generic (PLEG): container finished" podID="15d8b426-a116-44db-ae14-3242e17611b4" containerID="94dba0fe136cd70509580e116796bd1d107c89b43b5cf75bafdac1db45f85f6e" exitCode=0 Sep 16 14:15:40.001020 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:40.000967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" event={"ID":"15d8b426-a116-44db-ae14-3242e17611b4","Type":"ContainerDied","Data":"94dba0fe136cd70509580e116796bd1d107c89b43b5cf75bafdac1db45f85f6e"} Sep 16 14:15:40.002894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:40.002878 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" Sep 16 14:15:40.002995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:40.002912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apnrlw" event={"ID":"f043c50a-d9a4-4310-a3a3-794afee677b2","Type":"ContainerDied","Data":"9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f"} Sep 16 14:15:40.002995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:40.002945 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dff099cf28d788ab528da986e80399f6d7257b339df28bee16521aa7f968c4f" Sep 16 14:15:41.148740 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.148715 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:41.190204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.190180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:41.193284 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.193265 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:41.291910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291827 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util\") pod \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " Sep 16 14:15:41.291910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util\") pod \"15d8b426-a116-44db-ae14-3242e17611b4\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " Sep 16 14:15:41.291910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291899 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gbh7\" (UniqueName: \"kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7\") pod \"15d8b426-a116-44db-ae14-3242e17611b4\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291919 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle\") pod \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291978 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4gl\" (UniqueName: \"kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl\") pod \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\" (UID: \"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.291993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util\") pod \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle\") pod \"15d8b426-a116-44db-ae14-3242e17611b4\" (UID: \"15d8b426-a116-44db-ae14-3242e17611b4\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292040 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvdsx\" (UniqueName: \"kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx\") pod \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " Sep 16 14:15:41.292207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292074 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle\") pod \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\" (UID: \"1d9745b4-e14c-4cb8-aefb-a794375f28cf\") " Sep 16 14:15:41.292834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292640 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle" (OuterVolumeSpecName: "bundle") pod "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" (UID: "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.292834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle" (OuterVolumeSpecName: "bundle") pod "1d9745b4-e14c-4cb8-aefb-a794375f28cf" (UID: "1d9745b4-e14c-4cb8-aefb-a794375f28cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.293079 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.292864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle" (OuterVolumeSpecName: "bundle") pod "15d8b426-a116-44db-ae14-3242e17611b4" (UID: "15d8b426-a116-44db-ae14-3242e17611b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.294879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.294831 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl" (OuterVolumeSpecName: "kube-api-access-st4gl") pod "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" (UID: "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1"). InnerVolumeSpecName "kube-api-access-st4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:41.294879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.294874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7" (OuterVolumeSpecName: "kube-api-access-2gbh7") pod "15d8b426-a116-44db-ae14-3242e17611b4" (UID: "15d8b426-a116-44db-ae14-3242e17611b4"). InnerVolumeSpecName "kube-api-access-2gbh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:41.295231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.295207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx" (OuterVolumeSpecName: "kube-api-access-lvdsx") pod "1d9745b4-e14c-4cb8-aefb-a794375f28cf" (UID: "1d9745b4-e14c-4cb8-aefb-a794375f28cf"). InnerVolumeSpecName "kube-api-access-lvdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:41.298195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.298168 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util" (OuterVolumeSpecName: "util") pod "1d9745b4-e14c-4cb8-aefb-a794375f28cf" (UID: "1d9745b4-e14c-4cb8-aefb-a794375f28cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.298638 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.298595 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util" (OuterVolumeSpecName: "util") pod "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" (UID: "1372bfb4-3967-4030-84cb-5f6cfd5bb3a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.298742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.298722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util" (OuterVolumeSpecName: "util") pod "15d8b426-a116-44db-ae14-3242e17611b4" (UID: "15d8b426-a116-44db-ae14-3242e17611b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:41.392710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392661 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.392710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392702 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbh7\" (UniqueName: \"kubernetes.io/projected/15d8b426-a116-44db-ae14-3242e17611b4-kube-api-access-2gbh7\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.392710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392712 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.392710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392721 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4gl\" (UniqueName: \"kubernetes.io/projected/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-kube-api-access-st4gl\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.392710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392730 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.393011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392738 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d8b426-a116-44db-ae14-3242e17611b4-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.393011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392746 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdsx\" (UniqueName: \"kubernetes.io/projected/1d9745b4-e14c-4cb8-aefb-a794375f28cf-kube-api-access-lvdsx\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.393011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392761 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d9745b4-e14c-4cb8-aefb-a794375f28cf-bundle\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:41.393011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:41.392769 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1372bfb4-3967-4030-84cb-5f6cfd5bb3a1-util\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:15:42.015033 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.015001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" Sep 16 14:15:42.015188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.015023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78bgq5" event={"ID":"1d9745b4-e14c-4cb8-aefb-a794375f28cf","Type":"ContainerDied","Data":"161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390"} Sep 16 14:15:42.015188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.015083 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161f5383c4f1deb40a3a7db4813b859793db26dbf29371d3c072b35f082ca390" Sep 16 14:15:42.017124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.017066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" event={"ID":"1372bfb4-3967-4030-84cb-5f6cfd5bb3a1","Type":"ContainerDied","Data":"fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a"} Sep 16 14:15:42.017124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.017107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv52g" Sep 16 14:15:42.017329 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.017109 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac574003a3a8c3087669a88e57331a594b80365793999646faea18b7152313a" Sep 16 14:15:42.019297 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.019273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" event={"ID":"15d8b426-a116-44db-ae14-3242e17611b4","Type":"ContainerDied","Data":"913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338"} Sep 16 14:15:42.019396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.019309 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913fed1c09962017544a37bef530bfa67e74a76f7990e11607c4313ec5bca338" Sep 16 14:15:42.019396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:42.019387 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jvbz9" Sep 16 14:15:45.441030 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:45.440990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:46.233447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:46.233409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:48.716852 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.716822 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jdml5"] Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717142 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="pull" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717154 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="pull" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717162 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="pull" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717167 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="pull" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717176 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="util" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717181 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="util" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717188 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="extract" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717193 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="extract" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717203 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="util" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717208 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="util" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717217 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="util" Sep 16 14:15:48.717215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717222 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="util" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717227 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717233 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717241 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="pull" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717246 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="pull" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717251 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717256 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717265 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="util" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717270 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="util" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717292 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="pull" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717297 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="pull" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717304 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717310 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717359 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d8b426-a116-44db-ae14-3242e17611b4" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717367 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f043c50a-d9a4-4310-a3a3-794afee677b2" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717372 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1372bfb4-3967-4030-84cb-5f6cfd5bb3a1" containerName="extract" Sep 16 14:15:48.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.717378 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9745b4-e14c-4cb8-aefb-a794375f28cf" containerName="extract" Sep 16 14:15:48.720411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.720396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:48.723022 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.722990 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 14:15:48.723162 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.723056 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8lh4r\"" Sep 16 14:15:48.723354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.723337 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 14:15:48.733363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.733342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jdml5"] Sep 16 14:15:48.747818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.747796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2std\" (UniqueName: \"kubernetes.io/projected/55424c4e-8138-48b8-8777-e93bdc92b45f-kube-api-access-q2std\") pod \"authorino-operator-76b478bd78-jdml5\" (UID: \"55424c4e-8138-48b8-8777-e93bdc92b45f\") " pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:48.848442 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.848415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2std\" (UniqueName: \"kubernetes.io/projected/55424c4e-8138-48b8-8777-e93bdc92b45f-kube-api-access-q2std\") pod \"authorino-operator-76b478bd78-jdml5\" (UID: \"55424c4e-8138-48b8-8777-e93bdc92b45f\") " pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:48.858094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:48.858072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2std\" (UniqueName: \"kubernetes.io/projected/55424c4e-8138-48b8-8777-e93bdc92b45f-kube-api-access-q2std\") pod \"authorino-operator-76b478bd78-jdml5\" (UID: \"55424c4e-8138-48b8-8777-e93bdc92b45f\") " pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:49.030309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:49.030238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:49.311654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:49.311560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jdml5"] Sep 16 14:15:49.314919 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:49.314889 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55424c4e_8138_48b8_8777_e93bdc92b45f.slice/crio-a7ebae60ccc54604b5678e0adc380aa21dc39b4c062cf18a93cd272282b5685d WatchSource:0}: Error finding container a7ebae60ccc54604b5678e0adc380aa21dc39b4c062cf18a93cd272282b5685d: Status 404 returned error can't find the container with id a7ebae60ccc54604b5678e0adc380aa21dc39b4c062cf18a93cd272282b5685d Sep 16 14:15:49.333779 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:49.333753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:49.451747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:49.451713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:49.556665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:49.556624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:50.067324 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:50.067285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" event={"ID":"55424c4e-8138-48b8-8777-e93bdc92b45f","Type":"ContainerStarted","Data":"a7ebae60ccc54604b5678e0adc380aa21dc39b4c062cf18a93cd272282b5685d"} Sep 16 14:15:51.087991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.087961 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp"] Sep 16 14:15:51.092193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.092171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:51.095556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.095517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h8bft\"" Sep 16 14:15:51.116981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.116952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp"] Sep 16 14:15:51.167856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.167817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgn7\" (UniqueName: \"kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7\") pod \"limitador-operator-controller-manager-74495c69f8-qg7pp\" (UID: \"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:51.269155 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.269117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgn7\" (UniqueName: \"kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7\") pod \"limitador-operator-controller-manager-74495c69f8-qg7pp\" (UID: \"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:51.278091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.278066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgn7\" (UniqueName: \"kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7\") pod \"limitador-operator-controller-manager-74495c69f8-qg7pp\" (UID: \"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:51.404857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:51.404778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:52.079252 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.079192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:52.080477 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.080455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" event={"ID":"55424c4e-8138-48b8-8777-e93bdc92b45f","Type":"ContainerStarted","Data":"e2d352938ef067e166797d595ee7def84ed6e31c3b4ee7c39281083286ba93ba"} Sep 16 14:15:52.080651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.080631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:15:52.100705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.100663 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" podStartSLOduration=1.432193753 podStartE2EDuration="4.100648993s" totalImagesPullingTime="2.66845524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:52.098353309 +0000 UTC m=+482.675714462" watchObservedRunningTime="2026-09-16 14:15:52.100648993 +0000 UTC m=+482.678010146" Sep 16 14:15:52.180347 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.180318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:52.188667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:52.188648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp"] Sep 16 14:15:52.190728 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:52.190703 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5da957_e30a_4d8d_8f49_dbd42e74b7d2.slice/crio-8499a0cd94e2f78ed7174581227af50f6aaa6c7aa52e66cb86aa6d5f2d18acec WatchSource:0}: Error finding container 8499a0cd94e2f78ed7174581227af50f6aaa6c7aa52e66cb86aa6d5f2d18acec: Status 404 returned error can't find the container with id 8499a0cd94e2f78ed7174581227af50f6aaa6c7aa52e66cb86aa6d5f2d18acec Sep 16 14:15:53.087743 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.087703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" event={"ID":"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2","Type":"ContainerStarted","Data":"8499a0cd94e2f78ed7174581227af50f6aaa6c7aa52e66cb86aa6d5f2d18acec"} Sep 16 14:15:53.329891 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.329858 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5"] Sep 16 14:15:53.333780 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.333757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:53.339209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.338824 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6x5qp\"" Sep 16 14:15:53.339209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.338824 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 16 14:15:53.378931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.378901 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5"] Sep 16 14:15:53.385361 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.385333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p977d\" (UniqueName: \"kubernetes.io/projected/6acdab5a-a4cd-45d3-84a0-cc23194d56eb-kube-api-access-p977d\") pod \"dns-operator-controller-manager-7b84cd7c9c-8ghb5\" (UID: \"6acdab5a-a4cd-45d3-84a0-cc23194d56eb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:53.486502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.486457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p977d\" (UniqueName: \"kubernetes.io/projected/6acdab5a-a4cd-45d3-84a0-cc23194d56eb-kube-api-access-p977d\") pod \"dns-operator-controller-manager-7b84cd7c9c-8ghb5\" (UID: \"6acdab5a-a4cd-45d3-84a0-cc23194d56eb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:53.509306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.509271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p977d\" (UniqueName: \"kubernetes.io/projected/6acdab5a-a4cd-45d3-84a0-cc23194d56eb-kube-api-access-p977d\") pod \"dns-operator-controller-manager-7b84cd7c9c-8ghb5\" (UID: \"6acdab5a-a4cd-45d3-84a0-cc23194d56eb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:53.647306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.647274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:53.896423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.896307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:53.939037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:53.939004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5"] Sep 16 14:15:53.942033 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:53.942007 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acdab5a_a4cd_45d3_84a0_cc23194d56eb.slice/crio-d764d230509565d52342dfb90f18474953be04b9d2e9f35f7e42fcb82a187b47 WatchSource:0}: Error finding container d764d230509565d52342dfb90f18474953be04b9d2e9f35f7e42fcb82a187b47: Status 404 returned error can't find the container with id d764d230509565d52342dfb90f18474953be04b9d2e9f35f7e42fcb82a187b47 Sep 16 14:15:54.022342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.022308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:54.094899 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.094862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" event={"ID":"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2","Type":"ContainerStarted","Data":"da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5"} Sep 16 14:15:54.095101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.094959 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:15:54.096317 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.096295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" event={"ID":"6acdab5a-a4cd-45d3-84a0-cc23194d56eb","Type":"ContainerStarted","Data":"d764d230509565d52342dfb90f18474953be04b9d2e9f35f7e42fcb82a187b47"} Sep 16 14:15:54.125274 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.125215 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" podStartSLOduration=1.759884408 podStartE2EDuration="3.125199565s" totalImagesPullingTime="1.365315157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:54.123721759 +0000 UTC m=+484.701082911" watchObservedRunningTime="2026-09-16 14:15:54.125199565 +0000 UTC m=+484.702560717" Sep 16 14:15:54.729182 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.729145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:54.966698 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:54.966659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:55.453299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:55.453131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:56.111016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:56.110935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" event={"ID":"6acdab5a-a4cd-45d3-84a0-cc23194d56eb","Type":"ContainerStarted","Data":"d64e2348f6369a4d58ded4c7cd43f82ee26c6cda1a6a4b09d4ca3cb67e6c6251"} Sep 16 14:15:56.111406 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:56.111118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:15:56.141550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:56.141498 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" podStartSLOduration=1.4070785049999999 podStartE2EDuration="3.141482781s" totalImagesPullingTime="1.734404276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:56.138829 +0000 UTC m=+486.716190151" watchObservedRunningTime="2026-09-16 14:15:56.141482781 +0000 UTC m=+486.718843934" Sep 16 14:15:56.219704 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:56.219672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:58.310295 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.310258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx"] Sep 16 14:15:58.314139 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.314121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.321883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.321864 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pnf42\"" Sep 16 14:15:58.356149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.356123 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx"] Sep 16 14:15:58.430008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.429979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7fh\" (UniqueName: \"kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.430146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.430014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.531342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.531307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7fh\" (UniqueName: \"kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.531489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.531351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.531762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.531740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.558447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.558420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7fh\" (UniqueName: \"kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.624298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.624219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:15:58.930700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:58.930673 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx"] Sep 16 14:15:58.931721 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:15:58.931692 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e0918a_7bb2_4608_9e63_c048d2e45495.slice/crio-117f768f0f1ee7515d67cf35c69bf10d71178387d271f1c33e8f5813e1563fcf WatchSource:0}: Error finding container 117f768f0f1ee7515d67cf35c69bf10d71178387d271f1c33e8f5813e1563fcf: Status 404 returned error can't find the container with id 117f768f0f1ee7515d67cf35c69bf10d71178387d271f1c33e8f5813e1563fcf Sep 16 14:15:59.135144 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:59.135114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" event={"ID":"a3e0918a-7bb2-4608-9e63-c048d2e45495","Type":"ContainerStarted","Data":"117f768f0f1ee7515d67cf35c69bf10d71178387d271f1c33e8f5813e1563fcf"} Sep 16 14:15:59.684342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:59.684311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:59.786226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:15:59.786196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:00.175477 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:00.175430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:00.830661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:00.830594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:03.090487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.090404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-jdml5" Sep 16 14:16:03.162796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.162758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" event={"ID":"a3e0918a-7bb2-4608-9e63-c048d2e45495","Type":"ContainerStarted","Data":"6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411"} Sep 16 14:16:03.162978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.162908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:03.212225 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.212173 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" podStartSLOduration=1.358178506 podStartE2EDuration="5.212156949s" totalImagesPullingTime="3.853978443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:03.207689073 +0000 UTC m=+493.785050224" watchObservedRunningTime="2026-09-16 14:16:03.212156949 +0000 UTC m=+493.789518101" Sep 16 14:16:03.651489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.651448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:03.774367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.774334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:03.904820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.904729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:03.999826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:03.999784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:05.105961 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:05.105925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:05.584868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:05.584832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:05.806024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:05.805988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:05.920686 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:05.920576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:06.257656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:06.257624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:06.380031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:06.379998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:07.119741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:07.119711 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8ghb5" Sep 16 14:16:11.793759 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:11.793725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:11.906659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:11.906626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:12.031242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:12.031216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:14.170577 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:14.170544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:15.188762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.188727 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh"] Sep 16 14:16:15.192024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.192007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.223895 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.223868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh"] Sep 16 14:16:15.271368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.271328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.271543 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.271461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crlh\" (UniqueName: \"kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.372151 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.372105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crlh\" (UniqueName: \"kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.372358 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.372166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.372552 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.372533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.393700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.393678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crlh\" (UniqueName: \"kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.502137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.502049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:15.788529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:15.788505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh"] Sep 16 14:16:15.790954 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:16:15.790922 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556ce7f2_66c5_4252_b973_e2a1404d465b.slice/crio-b75cec411f9eaf0e18b99e35079246d9da60c7e7d4b97db10072b77ed3a17321 WatchSource:0}: Error finding container b75cec411f9eaf0e18b99e35079246d9da60c7e7d4b97db10072b77ed3a17321: Status 404 returned error can't find the container with id b75cec411f9eaf0e18b99e35079246d9da60c7e7d4b97db10072b77ed3a17321 Sep 16 14:16:16.237801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.237759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" event={"ID":"556ce7f2-66c5-4252-b973-e2a1404d465b","Type":"ContainerStarted","Data":"7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863"} Sep 16 14:16:16.238271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.237825 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" event={"ID":"556ce7f2-66c5-4252-b973-e2a1404d465b","Type":"ContainerStarted","Data":"b75cec411f9eaf0e18b99e35079246d9da60c7e7d4b97db10072b77ed3a17321"} Sep 16 14:16:16.238271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.237846 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:16.269515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.269463 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" podStartSLOduration=1.2694476319999999 podStartE2EDuration="1.269447632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:16.268386765 +0000 UTC m=+506.845747924" watchObservedRunningTime="2026-09-16 14:16:16.269447632 +0000 UTC m=+506.846808783" Sep 16 14:16:16.331728 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.331684 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh"] Sep 16 14:16:16.341689 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.341653 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh"] Sep 16 14:16:16.357316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.357279 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx"] Sep 16 14:16:16.357510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.357472 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" containerName="manager" containerID="cri-o://6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411" gracePeriod=2 Sep 16 14:16:16.363112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.363077 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:16.363484 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.363470 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" containerName="manager" Sep 16 14:16:16.363484 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.363485 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" containerName="manager" Sep 16 14:16:16.363620 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.363539 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" containerName="manager" Sep 16 14:16:16.367148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.366961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.372934 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.372876 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx"] Sep 16 14:16:16.390667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.390553 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp"] Sep 16 14:16:16.390904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.390877 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" containerName="manager" containerID="cri-o://da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5" gracePeriod=2 Sep 16 14:16:16.393661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.393638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:16.404012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.403988 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp"] Sep 16 14:16:16.406579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.406557 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:16.407071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407051 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" containerName="manager" Sep 16 14:16:16.407170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407074 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" containerName="manager" Sep 16 14:16:16.407170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407089 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" containerName="manager" Sep 16 14:16:16.407170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407094 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" containerName="manager" Sep 16 14:16:16.407376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407183 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" containerName="manager" Sep 16 14:16:16.407376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.407194 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" containerName="manager" Sep 16 14:16:16.410313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.410295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.430088 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.430051 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.432836 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.432693 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.434716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.434692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:16.441042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.441019 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k"] Sep 16 14:16:16.444987 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.444966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:16.465546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.465505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k"] Sep 16 14:16:16.473772 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.473732 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:16.482176 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.482150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8kl\" (UniqueName: \"kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.482280 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.482212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.482280 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.482268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmpl\" (UniqueName: \"kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.482367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.482343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.482549 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.482528 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.511942 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.511906 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:16.583298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8kl\" (UniqueName: \"kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.583453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.583453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxs9\" (UniqueName: \"kubernetes.io/projected/d5c49d6e-3021-4281-9eb7-e64357ab3dfa-kube-api-access-dcxs9\") pod \"limitador-operator-controller-manager-74495c69f8-fzq7k\" (UID: \"d5c49d6e-3021-4281-9eb7-e64357ab3dfa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:16.583453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmpl\" (UniqueName: \"kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.584039 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.584039 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.584039 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.583902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.596063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.596041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8kl\" (UniqueName: \"kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-nz6vx\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.598418 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.598398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmpl\" (UniqueName: \"kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gtkvj\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.654264 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.654240 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.657320 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.657297 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:16.658534 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.658512 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.661298 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.661275 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:16.663411 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.663387 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:16.665274 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:16.665255 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:16.684453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.684430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxs9\" (UniqueName: \"kubernetes.io/projected/d5c49d6e-3021-4281-9eb7-e64357ab3dfa-kube-api-access-dcxs9\") pod \"limitador-operator-controller-manager-74495c69f8-fzq7k\" (UID: \"d5c49d6e-3021-4281-9eb7-e64357ab3dfa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:16.711910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.711886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxs9\" (UniqueName: \"kubernetes.io/projected/d5c49d6e-3021-4281-9eb7-e64357ab3dfa-kube-api-access-dcxs9\") pod \"limitador-operator-controller-manager-74495c69f8-fzq7k\" (UID: \"d5c49d6e-3021-4281-9eb7-e64357ab3dfa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:16.774992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.774934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:16.780818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.780800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:16.785658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.785637 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc7fh\" (UniqueName: \"kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh\") pod \"a3e0918a-7bb2-4608-9e63-c048d2e45495\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " Sep 16 14:16:16.785756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.785721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwgn7\" (UniqueName: \"kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7\") pod \"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2\" (UID: \"ac5da957-e30a-4d8d-8f49-dbd42e74b7d2\") " Sep 16 14:16:16.785812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.785796 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume\") pod \"a3e0918a-7bb2-4608-9e63-c048d2e45495\" (UID: \"a3e0918a-7bb2-4608-9e63-c048d2e45495\") " Sep 16 14:16:16.786458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.786363 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a3e0918a-7bb2-4608-9e63-c048d2e45495" (UID: "a3e0918a-7bb2-4608-9e63-c048d2e45495"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:16:16.788073 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.788048 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh" (OuterVolumeSpecName: "kube-api-access-gc7fh") pod "a3e0918a-7bb2-4608-9e63-c048d2e45495" (UID: "a3e0918a-7bb2-4608-9e63-c048d2e45495"). InnerVolumeSpecName "kube-api-access-gc7fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:16:16.788073 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.788062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:16.788207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.788119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7" (OuterVolumeSpecName: "kube-api-access-dwgn7") pod "ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" (UID: "ac5da957-e30a-4d8d-8f49-dbd42e74b7d2"). InnerVolumeSpecName "kube-api-access-dwgn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:16:16.887277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.887252 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc7fh\" (UniqueName: \"kubernetes.io/projected/a3e0918a-7bb2-4608-9e63-c048d2e45495-kube-api-access-gc7fh\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:16.887277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.887277 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgn7\" (UniqueName: \"kubernetes.io/projected/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2-kube-api-access-dwgn7\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:16.887419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:16.887290 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e0918a-7bb2-4608-9e63-c048d2e45495-extensions-socket-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:17.113927 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.113886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:17.116969 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:16:17.116940 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42a17e4_18ff_4ae9_ac13_685cdd98c14c.slice/crio-85fb34dd002b427271a271b571220659d119691b89ebb754ef40644147097cda WatchSource:0}: Error finding container 85fb34dd002b427271a271b571220659d119691b89ebb754ef40644147097cda: Status 404 returned error can't find the container with id 85fb34dd002b427271a271b571220659d119691b89ebb754ef40644147097cda Sep 16 14:16:17.118780 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.118749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:17.119464 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:16:17.119439 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c49d6e_3021_4281_9eb7_e64357ab3dfa.slice/crio-bf04bf0d9eaa15ced2d8b444965c4799a337dc17d338498a50516f77883b6d06 WatchSource:0}: Error finding container bf04bf0d9eaa15ced2d8b444965c4799a337dc17d338498a50516f77883b6d06: Status 404 returned error can't find the container with id bf04bf0d9eaa15ced2d8b444965c4799a337dc17d338498a50516f77883b6d06 Sep 16 14:16:17.119722 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:16:17.119701 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf973133_c823_44b9_8ffd_b409b0fd5917.slice/crio-6d84d33d36197d2354aba1d36a0d912b5f987811f724cfcc31aa27a6de6faf3f WatchSource:0}: Error finding container 6d84d33d36197d2354aba1d36a0d912b5f987811f724cfcc31aa27a6de6faf3f: Status 404 returned error can't find the container with id 6d84d33d36197d2354aba1d36a0d912b5f987811f724cfcc31aa27a6de6faf3f Sep 16 14:16:17.123520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.123496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k"] Sep 16 14:16:17.250257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.250225 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" containerID="da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5" exitCode=0 Sep 16 14:16:17.250776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.250286 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:17.250776 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.250319 2579 scope.go:117] "RemoveContainer" containerID="da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5" Sep 16 14:16:17.251910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.251887 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3e0918a-7bb2-4608-9e63-c048d2e45495" containerID="6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411" exitCode=0 Sep 16 14:16:17.252008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.251952 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:17.253673 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.253645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" event={"ID":"d5c49d6e-3021-4281-9eb7-e64357ab3dfa","Type":"ContainerStarted","Data":"8fb32a969b2528a18f21fefe5115119f6e6fbdf1cb9ccf35f7603fd64d455e38"} Sep 16 14:16:17.253785 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.253738 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:17.253785 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.253762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" event={"ID":"d5c49d6e-3021-4281-9eb7-e64357ab3dfa","Type":"ContainerStarted","Data":"bf04bf0d9eaa15ced2d8b444965c4799a337dc17d338498a50516f77883b6d06"} Sep 16 14:16:17.254302 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.254276 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:17.256122 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.256087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" event={"ID":"b42a17e4-18ff-4ae9-ac13-685cdd98c14c","Type":"ContainerStarted","Data":"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102"} Sep 16 14:16:17.256240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.256140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" event={"ID":"b42a17e4-18ff-4ae9-ac13-685cdd98c14c","Type":"ContainerStarted","Data":"85fb34dd002b427271a271b571220659d119691b89ebb754ef40644147097cda"} Sep 16 14:16:17.256240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.256169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:17.256240 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.256194 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:17.258249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.258153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" event={"ID":"bf973133-c823-44b9-8ffd-b409b0fd5917","Type":"ContainerStarted","Data":"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f"} Sep 16 14:16:17.258249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.258212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:17.258249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.258234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" event={"ID":"bf973133-c823-44b9-8ffd-b409b0fd5917","Type":"ContainerStarted","Data":"6d84d33d36197d2354aba1d36a0d912b5f987811f724cfcc31aa27a6de6faf3f"} Sep 16 14:16:17.258445 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.258314 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" container="manager" Sep 16 14:16:17.262081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.262064 2579 scope.go:117] "RemoveContainer" containerID="da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5" Sep 16 14:16:17.262356 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.262338 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5\": container with ID starting with da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5 not found: ID does not exist" containerID="da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5" Sep 16 14:16:17.262426 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.262362 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5"} err="failed to get container status \"da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5\": rpc error: code = NotFound desc = could not find container \"da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5\": container with ID starting with da0e1d2ac97c7927942a6e78f046a40479defb41bfa5d40b811167192d810ce5 not found: ID does not exist" Sep 16 14:16:17.262426 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.262379 2579 scope.go:117] "RemoveContainer" containerID="6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411" Sep 16 14:16:17.262841 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.262817 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:17.266815 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.266789 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:17.271293 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.271278 2579 scope.go:117] "RemoveContainer" containerID="6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411" Sep 16 14:16:17.271551 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.271530 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411\": container with ID starting with 6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411 not found: ID does not exist" containerID="6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411" Sep 16 14:16:17.271648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.271556 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411"} err="failed to get container status \"6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411\": rpc error: code = NotFound desc = could not find container \"6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411\": container with ID starting with 6cd885edd4dbbd88d3d27b9776c5cdef22a85333a10c04d32cd4a2f8f7b1d411 not found: ID does not exist" Sep 16 14:16:17.354234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.354179 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" podStartSLOduration=1.3541620079999999 podStartE2EDuration="1.354162008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:17.3190604 +0000 UTC m=+507.896421552" watchObservedRunningTime="2026-09-16 14:16:17.354162008 +0000 UTC m=+507.931523161" Sep 16 14:16:17.354955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.354921 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" podStartSLOduration=1.354911227 podStartE2EDuration="1.354911227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:17.352785129 +0000 UTC m=+507.930146283" watchObservedRunningTime="2026-09-16 14:16:17.354911227 +0000 UTC m=+507.932272378" Sep 16 14:16:17.355688 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.355662 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qg7pp\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qg7pp" Sep 16 14:16:17.384245 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.384169 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" podStartSLOduration=1.384154523 podStartE2EDuration="1.384154523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:17.382158361 +0000 UTC m=+507.959519512" watchObservedRunningTime="2026-09-16 14:16:17.384154523 +0000 UTC m=+507.961515674" Sep 16 14:16:17.388642 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.388589 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-29hwx\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-29hwx" Sep 16 14:16:17.390422 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:17.390402 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:17.817594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.817560 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e0918a-7bb2-4608-9e63-c048d2e45495" path="/var/lib/kubelet/pods/a3e0918a-7bb2-4608-9e63-c048d2e45495/volumes" Sep 16 14:16:17.817937 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:17.817923 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5da957-e30a-4d8d-8f49-dbd42e74b7d2" path="/var/lib/kubelet/pods/ac5da957-e30a-4d8d-8f49-dbd42e74b7d2/volumes" Sep 16 14:16:18.268567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.268537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" containerName="manager" containerID="cri-o://7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863" gracePeriod=2 Sep 16 14:16:18.514643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.514596 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:18.604014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.603927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume\") pod \"556ce7f2-66c5-4252-b973-e2a1404d465b\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " Sep 16 14:16:18.604014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.603971 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7crlh\" (UniqueName: \"kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh\") pod \"556ce7f2-66c5-4252-b973-e2a1404d465b\" (UID: \"556ce7f2-66c5-4252-b973-e2a1404d465b\") " Sep 16 14:16:18.604341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.604316 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "556ce7f2-66c5-4252-b973-e2a1404d465b" (UID: "556ce7f2-66c5-4252-b973-e2a1404d465b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:16:18.606286 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.606253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh" (OuterVolumeSpecName: "kube-api-access-7crlh") pod "556ce7f2-66c5-4252-b973-e2a1404d465b" (UID: "556ce7f2-66c5-4252-b973-e2a1404d465b"). InnerVolumeSpecName "kube-api-access-7crlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:16:18.704830 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.704796 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/556ce7f2-66c5-4252-b973-e2a1404d465b-extensions-socket-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:18.704830 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:18.704822 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crlh\" (UniqueName: \"kubernetes.io/projected/556ce7f2-66c5-4252-b973-e2a1404d465b-kube-api-access-7crlh\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:19.276771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.276734 2579 generic.go:356] "Generic (PLEG): container finished" podID="556ce7f2-66c5-4252-b973-e2a1404d465b" containerID="7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863" exitCode=0 Sep 16 14:16:19.277247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.276790 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:19.277247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.276831 2579 scope.go:117] "RemoveContainer" containerID="7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863" Sep 16 14:16:19.279311 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:19.279284 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:19.286957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.286944 2579 scope.go:117] "RemoveContainer" containerID="7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863" Sep 16 14:16:19.287221 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:19.287202 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863\": container with ID starting with 7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863 not found: ID does not exist" containerID="7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863" Sep 16 14:16:19.287265 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.287232 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863"} err="failed to get container status \"7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863\": rpc error: code = NotFound desc = could not find container \"7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863\": container with ID starting with 7e546208186f8b8d15addd1db064e776fbc2c8ce8d4e3bc781c04844e5bfb863 not found: ID does not exist" Sep 16 14:16:19.288046 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:19.288030 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:19.816659 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:19.816580 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-zccsh\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-zccsh" Sep 16 14:16:19.818233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.818208 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="556ce7f2-66c5-4252-b973-e2a1404d465b" path="/var/lib/kubelet/pods/556ce7f2-66c5-4252-b973-e2a1404d465b/volumes" Sep 16 14:16:19.843826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.843790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:19.943781 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:19.943745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:20.461245 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:20.461212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:20.569446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:20.569413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:23.828525 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:23.828490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:23.932499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:23.932471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:26.040082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:26.040045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:26.141775 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:26.141736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:28.271135 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.271102 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fzq7k" Sep 16 14:16:28.271568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.271176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:28.271568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.271345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:28.382722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.382686 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:28.382946 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.382919 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" podUID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" containerName="manager" containerID="cri-o://c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102" gracePeriod=10 Sep 16 14:16:28.638002 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.637983 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:28.790553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.790525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume\") pod \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " Sep 16 14:16:28.790729 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.790561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqmpl\" (UniqueName: \"kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl\") pod \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\" (UID: \"b42a17e4-18ff-4ae9-ac13-685cdd98c14c\") " Sep 16 14:16:28.790919 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.790899 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b42a17e4-18ff-4ae9-ac13-685cdd98c14c" (UID: "b42a17e4-18ff-4ae9-ac13-685cdd98c14c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:16:28.792515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.792488 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl" (OuterVolumeSpecName: "kube-api-access-vqmpl") pod "b42a17e4-18ff-4ae9-ac13-685cdd98c14c" (UID: "b42a17e4-18ff-4ae9-ac13-685cdd98c14c"). InnerVolumeSpecName "kube-api-access-vqmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:16:28.891949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.891869 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-extensions-socket-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:28.891949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.891898 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmpl\" (UniqueName: \"kubernetes.io/projected/b42a17e4-18ff-4ae9-ac13-685cdd98c14c-kube-api-access-vqmpl\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:28.994269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.994238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:16:28.994600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.994588 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" containerName="manager" Sep 16 14:16:28.994600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.994615 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" containerName="manager" Sep 16 14:16:28.994713 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.994704 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" containerName="manager" Sep 16 14:16:28.998237 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:28.998222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.019208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.019183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:16:29.094335 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.094306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wq4\" (UniqueName: \"kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.094470 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.094344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.194850 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.194828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wq4\" (UniqueName: \"kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.195001 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.194865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.195203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.195187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.209206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.209186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wq4\" (UniqueName: \"kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8rjtb\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.308369 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.308344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:29.344200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.344163 2579 generic.go:356] "Generic (PLEG): container finished" podID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" containerID="c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102" exitCode=0 Sep 16 14:16:29.344342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.344211 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" Sep 16 14:16:29.344342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.344234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" event={"ID":"b42a17e4-18ff-4ae9-ac13-685cdd98c14c","Type":"ContainerDied","Data":"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102"} Sep 16 14:16:29.344342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.344288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj" event={"ID":"b42a17e4-18ff-4ae9-ac13-685cdd98c14c","Type":"ContainerDied","Data":"85fb34dd002b427271a271b571220659d119691b89ebb754ef40644147097cda"} Sep 16 14:16:29.344342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.344311 2579 scope.go:117] "RemoveContainer" containerID="c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102" Sep 16 14:16:29.355011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.354938 2579 scope.go:117] "RemoveContainer" containerID="c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102" Sep 16 14:16:29.355318 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:29.355299 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102\": container with ID starting with c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102 not found: ID does not exist" containerID="c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102" Sep 16 14:16:29.355415 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.355327 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102"} err="failed to get container status \"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102\": rpc error: code = NotFound desc = could not find container \"c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102\": container with ID starting with c90976daed125ba027b2412879cdff0c422b436d4298bb5811d867a880495102 not found: ID does not exist" Sep 16 14:16:29.382155 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.382123 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:29.391079 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.391052 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gtkvj"] Sep 16 14:16:29.591101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.591073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:16:29.592947 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:16:29.592919 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46627e4a_0375_45c1_bdbf_1731f8ca4df2.slice/crio-51b4e1e54984c0f6d19b2ece090c84384584529a4d0a6d8393cffc7387d48e81 WatchSource:0}: Error finding container 51b4e1e54984c0f6d19b2ece090c84384584529a4d0a6d8393cffc7387d48e81: Status 404 returned error can't find the container with id 51b4e1e54984c0f6d19b2ece090c84384584529a4d0a6d8393cffc7387d48e81 Sep 16 14:16:29.822207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:29.822171 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42a17e4-18ff-4ae9-ac13-685cdd98c14c" path="/var/lib/kubelet/pods/b42a17e4-18ff-4ae9-ac13-685cdd98c14c/volumes" Sep 16 14:16:30.352569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:30.352535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" event={"ID":"46627e4a-0375-45c1-bdbf-1731f8ca4df2","Type":"ContainerStarted","Data":"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad"} Sep 16 14:16:30.352955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:30.352585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" event={"ID":"46627e4a-0375-45c1-bdbf-1731f8ca4df2","Type":"ContainerStarted","Data":"51b4e1e54984c0f6d19b2ece090c84384584529a4d0a6d8393cffc7387d48e81"} Sep 16 14:16:30.352955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:30.352650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:30.379189 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:30.379148 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" podStartSLOduration=2.379134678 podStartE2EDuration="2.379134678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:30.377763137 +0000 UTC m=+520.955124312" watchObservedRunningTime="2026-09-16 14:16:30.379134678 +0000 UTC m=+520.956495875" Sep 16 14:16:31.941782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:31.941748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:32.050842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:32.050805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:32.137392 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:32.137363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:41.361104 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.361034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:16:41.422333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.422299 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:41.422705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.422510 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" podUID="bf973133-c823-44b9-8ffd-b409b0fd5917" containerName="manager" containerID="cri-o://c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f" gracePeriod=10 Sep 16 14:16:41.669089 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.669059 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:41.802502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.802475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume\") pod \"bf973133-c823-44b9-8ffd-b409b0fd5917\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " Sep 16 14:16:41.802700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.802514 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh8kl\" (UniqueName: \"kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl\") pod \"bf973133-c823-44b9-8ffd-b409b0fd5917\" (UID: \"bf973133-c823-44b9-8ffd-b409b0fd5917\") " Sep 16 14:16:41.802898 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.802874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bf973133-c823-44b9-8ffd-b409b0fd5917" (UID: "bf973133-c823-44b9-8ffd-b409b0fd5917"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:16:41.804808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.804788 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl" (OuterVolumeSpecName: "kube-api-access-jh8kl") pod "bf973133-c823-44b9-8ffd-b409b0fd5917" (UID: "bf973133-c823-44b9-8ffd-b409b0fd5917"). InnerVolumeSpecName "kube-api-access-jh8kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:16:41.903245 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.903183 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf973133-c823-44b9-8ffd-b409b0fd5917-extensions-socket-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:41.903245 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:41.903203 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8kl\" (UniqueName: \"kubernetes.io/projected/bf973133-c823-44b9-8ffd-b409b0fd5917-kube-api-access-jh8kl\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:16:42.422361 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.422325 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf973133-c823-44b9-8ffd-b409b0fd5917" containerID="c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f" exitCode=0 Sep 16 14:16:42.422823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.422373 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" Sep 16 14:16:42.422823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.422407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" event={"ID":"bf973133-c823-44b9-8ffd-b409b0fd5917","Type":"ContainerDied","Data":"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f"} Sep 16 14:16:42.422823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.422462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx" event={"ID":"bf973133-c823-44b9-8ffd-b409b0fd5917","Type":"ContainerDied","Data":"6d84d33d36197d2354aba1d36a0d912b5f987811f724cfcc31aa27a6de6faf3f"} Sep 16 14:16:42.422823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.422488 2579 scope.go:117] "RemoveContainer" containerID="c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f" Sep 16 14:16:42.434202 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.434182 2579 scope.go:117] "RemoveContainer" containerID="c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f" Sep 16 14:16:42.434673 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:16:42.434646 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f\": container with ID starting with c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f not found: ID does not exist" containerID="c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f" Sep 16 14:16:42.434788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.434684 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f"} err="failed to get container status \"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f\": rpc error: code = NotFound desc = could not find container \"c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f\": container with ID starting with c5194c1f1f5f7ad3017fbf03d43f11b02d2e309adf9198c0ae5ed6506dec045f not found: ID does not exist" Sep 16 14:16:42.450167 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.450145 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:42.454286 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:42.454264 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-nz6vx"] Sep 16 14:16:43.816291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:43.816262 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf973133-c823-44b9-8ffd-b409b0fd5917" path="/var/lib/kubelet/pods/bf973133-c823-44b9-8ffd-b409b0fd5917/volumes" Sep 16 14:16:44.858099 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:44.858071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:45.220559 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:16:45.220522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:00.761287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.761246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:00.771988 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.771953 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:00.772808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.772782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf973133-c823-44b9-8ffd-b409b0fd5917" containerName="manager" Sep 16 14:17:00.772808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.772804 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf973133-c823-44b9-8ffd-b409b0fd5917" containerName="manager" Sep 16 14:17:00.773005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.772890 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf973133-c823-44b9-8ffd-b409b0fd5917" containerName="manager" Sep 16 14:17:00.777708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.777684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.792684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.792659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4hfdv\"" Sep 16 14:17:00.793820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.793804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 14:17:00.803317 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.803288 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:00.854199 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.854161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrlz\" (UniqueName: \"kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.854374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.854240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.861969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.861939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:00.907987 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.907950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:00.955163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.955126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrlz\" (UniqueName: \"kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.955346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.955205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.955868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.955848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:00.964952 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:00.964933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrlz\" (UniqueName: \"kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz\") pod \"limitador-limitador-8588749977-sd64f\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:01.088575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.088482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:01.376699 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.376668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:01.378559 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:01.378518 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6872e5_cc4f_4ab3_aa26_aea3f9bec163.slice/crio-6fecaae73b8449cc0e8a35c0628910642f0f71ba8182d233ae337060171ec7d2 WatchSource:0}: Error finding container 6fecaae73b8449cc0e8a35c0628910642f0f71ba8182d233ae337060171ec7d2: Status 404 returned error can't find the container with id 6fecaae73b8449cc0e8a35c0628910642f0f71ba8182d233ae337060171ec7d2 Sep 16 14:17:01.526054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.526023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" event={"ID":"df6872e5-cc4f-4ab3-aa26-aea3f9bec163","Type":"ContainerStarted","Data":"6fecaae73b8449cc0e8a35c0628910642f0f71ba8182d233ae337060171ec7d2"} Sep 16 14:17:01.655742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.655667 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:01.660209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.660189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:01.664174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.664151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-slcv4\"" Sep 16 14:17:01.670529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.670505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:01.703439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.703395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:01.763901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.763866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr\") pod \"authorino-7bcfd8f9cd-s6687\" (UID: \"c7f0fd4f-58fe-40d9-be36-67ce129f4345\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:01.801385 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.801346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:01.865064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.865028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr\") pod \"authorino-7bcfd8f9cd-s6687\" (UID: \"c7f0fd4f-58fe-40d9-be36-67ce129f4345\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:01.880943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.880915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr\") pod \"authorino-7bcfd8f9cd-s6687\" (UID: \"c7f0fd4f-58fe-40d9-be36-67ce129f4345\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:01.970048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:01.970014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:02.259797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:02.259727 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:02.262297 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:02.262270 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f0fd4f_58fe_40d9_be36_67ce129f4345.slice/crio-65debe4b414382c86af5701faa8cc404a900d49830acaa4b3cc9b4bb27098c1c WatchSource:0}: Error finding container 65debe4b414382c86af5701faa8cc404a900d49830acaa4b3cc9b4bb27098c1c: Status 404 returned error can't find the container with id 65debe4b414382c86af5701faa8cc404a900d49830acaa4b3cc9b4bb27098c1c Sep 16 14:17:02.535870 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:02.535784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" event={"ID":"c7f0fd4f-58fe-40d9-be36-67ce129f4345","Type":"ContainerStarted","Data":"65debe4b414382c86af5701faa8cc404a900d49830acaa4b3cc9b4bb27098c1c"} Sep 16 14:17:03.549262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:03.549222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" event={"ID":"df6872e5-cc4f-4ab3-aa26-aea3f9bec163","Type":"ContainerStarted","Data":"082eb4d8c0b943e8b3428dd6a420830e2c9da7cad0f06e0c8df2afad46a7bd3a"} Sep 16 14:17:03.549718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:03.549297 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:03.568584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:03.568536 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" podStartSLOduration=2.328328938 podStartE2EDuration="3.568520039s" totalImagesPullingTime="1.240191101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:03.56749603 +0000 UTC m=+554.144857197" watchObservedRunningTime="2026-09-16 14:17:03.568520039 +0000 UTC m=+554.145881191" Sep 16 14:17:04.557197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:04.557161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" event={"ID":"c7f0fd4f-58fe-40d9-be36-67ce129f4345","Type":"ContainerStarted","Data":"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4"} Sep 16 14:17:04.573421 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:04.573370 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" podStartSLOduration=2.01319303 podStartE2EDuration="3.573354609s" totalImagesPullingTime="1.560161579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:04.572114392 +0000 UTC m=+555.149475545" watchObservedRunningTime="2026-09-16 14:17:04.573354609 +0000 UTC m=+555.150715762" Sep 16 14:17:05.246512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:05.246470 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:06.569184 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:06.569127 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" podUID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" containerName="authorino" containerID="cri-o://eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4" gracePeriod=30 Sep 16 14:17:06.815435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:06.815409 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:06.909582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:06.909498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr\") pod \"c7f0fd4f-58fe-40d9-be36-67ce129f4345\" (UID: \"c7f0fd4f-58fe-40d9-be36-67ce129f4345\") " Sep 16 14:17:06.911659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:06.911634 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr" (OuterVolumeSpecName: "kube-api-access-t2bvr") pod "c7f0fd4f-58fe-40d9-be36-67ce129f4345" (UID: "c7f0fd4f-58fe-40d9-be36-67ce129f4345"). InnerVolumeSpecName "kube-api-access-t2bvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:07.010491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.010470 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bvr\" (UniqueName: \"kubernetes.io/projected/c7f0fd4f-58fe-40d9-be36-67ce129f4345-kube-api-access-t2bvr\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:07.576219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.576183 2579 generic.go:356] "Generic (PLEG): container finished" podID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" containerID="eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4" exitCode=0 Sep 16 14:17:07.576651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.576236 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" Sep 16 14:17:07.576651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.576238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" event={"ID":"c7f0fd4f-58fe-40d9-be36-67ce129f4345","Type":"ContainerDied","Data":"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4"} Sep 16 14:17:07.576651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.576279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6687" event={"ID":"c7f0fd4f-58fe-40d9-be36-67ce129f4345","Type":"ContainerDied","Data":"65debe4b414382c86af5701faa8cc404a900d49830acaa4b3cc9b4bb27098c1c"} Sep 16 14:17:07.576651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.576301 2579 scope.go:117] "RemoveContainer" containerID="eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4" Sep 16 14:17:07.586069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.586045 2579 scope.go:117] "RemoveContainer" containerID="eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4" Sep 16 14:17:07.586351 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:07.586333 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4\": container with ID starting with eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4 not found: ID does not exist" containerID="eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4" Sep 16 14:17:07.586403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.586362 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4"} err="failed to get container status \"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4\": rpc error: code = NotFound desc = could not find container \"eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4\": container with ID starting with eee76599a85848580272822b74fb02156821be4fb4944021f5e8745612da5aa4 not found: ID does not exist" Sep 16 14:17:07.597207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.597184 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:07.600894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.600862 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6687"] Sep 16 14:17:07.818192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:07.818160 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" path="/var/lib/kubelet/pods/c7f0fd4f-58fe-40d9-be36-67ce129f4345/volumes" Sep 16 14:17:14.558133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:14.558104 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:16.240962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:16.240920 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:16.241394 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:16.241115 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" podUID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" containerName="limitador" containerID="cri-o://082eb4d8c0b943e8b3428dd6a420830e2c9da7cad0f06e0c8df2afad46a7bd3a" gracePeriod=30 Sep 16 14:17:16.630996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:16.630965 2579 generic.go:356] "Generic (PLEG): container finished" podID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" containerID="082eb4d8c0b943e8b3428dd6a420830e2c9da7cad0f06e0c8df2afad46a7bd3a" exitCode=0 Sep 16 14:17:16.631142 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:16.631038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" event={"ID":"df6872e5-cc4f-4ab3-aa26-aea3f9bec163","Type":"ContainerDied","Data":"082eb4d8c0b943e8b3428dd6a420830e2c9da7cad0f06e0c8df2afad46a7bd3a"} Sep 16 14:17:17.181835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.181807 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:17.298419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.298386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxrlz\" (UniqueName: \"kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz\") pod \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " Sep 16 14:17:17.298909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.298440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file\") pod \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\" (UID: \"df6872e5-cc4f-4ab3-aa26-aea3f9bec163\") " Sep 16 14:17:17.298909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.298795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file" (OuterVolumeSpecName: "config-file") pod "df6872e5-cc4f-4ab3-aa26-aea3f9bec163" (UID: "df6872e5-cc4f-4ab3-aa26-aea3f9bec163"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:17:17.300795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.300772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz" (OuterVolumeSpecName: "kube-api-access-lxrlz") pod "df6872e5-cc4f-4ab3-aa26-aea3f9bec163" (UID: "df6872e5-cc4f-4ab3-aa26-aea3f9bec163"). InnerVolumeSpecName "kube-api-access-lxrlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:17.399764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.399736 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrlz\" (UniqueName: \"kubernetes.io/projected/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-kube-api-access-lxrlz\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:17.399764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.399759 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df6872e5-cc4f-4ab3-aa26-aea3f9bec163-config-file\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:17.637908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.637828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" event={"ID":"df6872e5-cc4f-4ab3-aa26-aea3f9bec163","Type":"ContainerDied","Data":"6fecaae73b8449cc0e8a35c0628910642f0f71ba8182d233ae337060171ec7d2"} Sep 16 14:17:17.637908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.637859 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sd64f" Sep 16 14:17:17.637908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.637903 2579 scope.go:117] "RemoveContainer" containerID="082eb4d8c0b943e8b3428dd6a420830e2c9da7cad0f06e0c8df2afad46a7bd3a" Sep 16 14:17:17.661996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.661971 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:17.664918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.664896 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sd64f"] Sep 16 14:17:17.817226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:17.817193 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" path="/var/lib/kubelet/pods/df6872e5-cc4f-4ab3-aa26-aea3f9bec163/volumes" Sep 16 14:17:19.056040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.056005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:19.156060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.156028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:19.515761 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.515732 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8x927"] Sep 16 14:17:19.516119 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516107 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" containerName="limitador" Sep 16 14:17:19.516160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516121 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" containerName="limitador" Sep 16 14:17:19.516160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516132 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" containerName="authorino" Sep 16 14:17:19.516160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516138 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" containerName="authorino" Sep 16 14:17:19.516254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516199 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6872e5-cc4f-4ab3-aa26-aea3f9bec163" containerName="limitador" Sep 16 14:17:19.516254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.516210 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f0fd4f-58fe-40d9-be36-67ce129f4345" containerName="authorino" Sep 16 14:17:19.520678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.520652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.523325 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.523300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-brqml\"" Sep 16 14:17:19.523451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.523431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 16 14:17:19.524205 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.524185 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:17:19.524308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.524267 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 14:17:19.538455 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.538432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8x927"] Sep 16 14:17:19.554997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.554961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:19.618696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.618663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bca2829f-723f-4d0f-957c-33984fc72b54-data\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.618868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.618712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzk7\" (UniqueName: \"kubernetes.io/projected/bca2829f-723f-4d0f-957c-33984fc72b54-kube-api-access-rpzk7\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.719918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.719880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bca2829f-723f-4d0f-957c-33984fc72b54-data\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.719918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.719918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzk7\" (UniqueName: \"kubernetes.io/projected/bca2829f-723f-4d0f-957c-33984fc72b54-kube-api-access-rpzk7\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.720263 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.720246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bca2829f-723f-4d0f-957c-33984fc72b54-data\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.729767 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.729744 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzk7\" (UniqueName: \"kubernetes.io/projected/bca2829f-723f-4d0f-957c-33984fc72b54-kube-api-access-rpzk7\") pod \"postgres-69fb4594cf-8x927\" (UID: \"bca2829f-723f-4d0f-957c-33984fc72b54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:19.830685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:19.830579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:20.106454 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:20.106426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8x927"] Sep 16 14:17:20.109044 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:20.109012 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2829f_723f_4d0f_957c_33984fc72b54.slice/crio-9852df3bd0024c404d745c634a9d8cd5fd8fe2f551e4f341c42f68bc015d70da WatchSource:0}: Error finding container 9852df3bd0024c404d745c634a9d8cd5fd8fe2f551e4f341c42f68bc015d70da: Status 404 returned error can't find the container with id 9852df3bd0024c404d745c634a9d8cd5fd8fe2f551e4f341c42f68bc015d70da Sep 16 14:17:20.657500 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:20.657465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" event={"ID":"bca2829f-723f-4d0f-957c-33984fc72b54","Type":"ContainerStarted","Data":"9852df3bd0024c404d745c634a9d8cd5fd8fe2f551e4f341c42f68bc015d70da"} Sep 16 14:17:25.690980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:25.690945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" event={"ID":"bca2829f-723f-4d0f-957c-33984fc72b54","Type":"ContainerStarted","Data":"359bd51dbcea0c79c57867f5c49fe1bb0b5772c1e36f80a987c0c0f3391cea2f"} Sep 16 14:17:25.691367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:25.691086 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:25.707885 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:25.707846 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" podStartSLOduration=1.51042569 podStartE2EDuration="6.707832264s" totalImagesPullingTime="5.197406574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:25.707167088 +0000 UTC m=+576.284528242" watchObservedRunningTime="2026-09-16 14:17:25.707832264 +0000 UTC m=+576.285193416" Sep 16 14:17:31.726029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:31.726000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8x927" Sep 16 14:17:32.069168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.069085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:32.259401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.259371 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rf2xb"] Sep 16 14:17:32.265787 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.265768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.268308 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.268289 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-slcv4\"" Sep 16 14:17:32.270377 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.270353 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rf2xb"] Sep 16 14:17:32.428541 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.428506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92ss\" (UniqueName: \"kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss\") pod \"authorino-557f876f69-rf2xb\" (UID: \"b8014e27-1afe-4190-a370-df834d5743ae\") " pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.462216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.462177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:32.483811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.483783 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rf2xb"] Sep 16 14:17:32.484050 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:32.484032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x92ss], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-rf2xb" podUID="b8014e27-1afe-4190-a370-df834d5743ae" Sep 16 14:17:32.513983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.513950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ffcc8c58-tk29k"] Sep 16 14:17:32.518460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.518442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.526729 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.526707 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ffcc8c58-tk29k"] Sep 16 14:17:32.529461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.529440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92ss\" (UniqueName: \"kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss\") pod \"authorino-557f876f69-rf2xb\" (UID: \"b8014e27-1afe-4190-a370-df834d5743ae\") " pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.543444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.543419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92ss\" (UniqueName: \"kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss\") pod \"authorino-557f876f69-rf2xb\" (UID: \"b8014e27-1afe-4190-a370-df834d5743ae\") " pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.551496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.551473 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffcc8c58-tk29k"] Sep 16 14:17:32.551760 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:32.551738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8cgqp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6ffcc8c58-tk29k" podUID="bd5dc432-0f01-4159-a198-9c285e053e39" Sep 16 14:17:32.589515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.589488 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:17:32.594113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.594092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.596645 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.596624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 14:17:32.602468 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.602446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:17:32.630446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.630415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgqp\" (UniqueName: \"kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp\") pod \"authorino-6ffcc8c58-tk29k\" (UID: \"bd5dc432-0f01-4159-a198-9c285e053e39\") " pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.729557 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.729473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.729929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.729475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.731280 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.731263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgqp\" (UniqueName: \"kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp\") pod \"authorino-6ffcc8c58-tk29k\" (UID: \"bd5dc432-0f01-4159-a198-9c285e053e39\") " pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.731341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.731309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56r7\" (UniqueName: \"kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.731341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.731330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.737260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.737241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.740376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.740361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:32.741695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.741675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgqp\" (UniqueName: \"kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp\") pod \"authorino-6ffcc8c58-tk29k\" (UID: \"bd5dc432-0f01-4159-a198-9c285e053e39\") " pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:32.831756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.831727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x92ss\" (UniqueName: \"kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss\") pod \"b8014e27-1afe-4190-a370-df834d5743ae\" (UID: \"b8014e27-1afe-4190-a370-df834d5743ae\") " Sep 16 14:17:32.831929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.831776 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cgqp\" (UniqueName: \"kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp\") pod \"bd5dc432-0f01-4159-a198-9c285e053e39\" (UID: \"bd5dc432-0f01-4159-a198-9c285e053e39\") " Sep 16 14:17:32.831929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.831908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56r7\" (UniqueName: \"kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.832063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.831929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.834102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.834077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp" (OuterVolumeSpecName: "kube-api-access-8cgqp") pod "bd5dc432-0f01-4159-a198-9c285e053e39" (UID: "bd5dc432-0f01-4159-a198-9c285e053e39"). InnerVolumeSpecName "kube-api-access-8cgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:32.834246 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.834215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss" (OuterVolumeSpecName: "kube-api-access-x92ss") pod "b8014e27-1afe-4190-a370-df834d5743ae" (UID: "b8014e27-1afe-4190-a370-df834d5743ae"). InnerVolumeSpecName "kube-api-access-x92ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:32.834490 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.834474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.840647 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.840597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56r7\" (UniqueName: \"kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7\") pod \"authorino-55bc75bc5b-65fq5\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.905621 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.905584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:17:32.933296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.933269 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92ss\" (UniqueName: \"kubernetes.io/projected/b8014e27-1afe-4190-a370-df834d5743ae-kube-api-access-x92ss\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:32.933296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:32.933297 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgqp\" (UniqueName: \"kubernetes.io/projected/bd5dc432-0f01-4159-a198-9c285e053e39-kube-api-access-8cgqp\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:33.181986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.181961 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:17:33.183718 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:33.183688 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod535eb2f1_6571_45c3_97b4_aeac035271f4.slice/crio-f24618a1dbd2ab7a32a162ec96057e08fd7e759656291675a8835191fd4b097b WatchSource:0}: Error finding container f24618a1dbd2ab7a32a162ec96057e08fd7e759656291675a8835191fd4b097b: Status 404 returned error can't find the container with id f24618a1dbd2ab7a32a162ec96057e08fd7e759656291675a8835191fd4b097b Sep 16 14:17:33.735913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.735889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" event={"ID":"535eb2f1-6571-45c3-97b4-aeac035271f4","Type":"ContainerStarted","Data":"f24618a1dbd2ab7a32a162ec96057e08fd7e759656291675a8835191fd4b097b"} Sep 16 14:17:33.736191 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.735954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffcc8c58-tk29k" Sep 16 14:17:33.736191 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.735957 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rf2xb" Sep 16 14:17:33.769163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.769140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rf2xb"] Sep 16 14:17:33.772960 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.772939 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rf2xb"] Sep 16 14:17:33.794521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.794496 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffcc8c58-tk29k"] Sep 16 14:17:33.801739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.801715 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6ffcc8c58-tk29k"] Sep 16 14:17:33.817892 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.817867 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8014e27-1afe-4190-a370-df834d5743ae" path="/var/lib/kubelet/pods/b8014e27-1afe-4190-a370-df834d5743ae/volumes" Sep 16 14:17:33.818095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:33.818084 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5dc432-0f01-4159-a198-9c285e053e39" path="/var/lib/kubelet/pods/bd5dc432-0f01-4159-a198-9c285e053e39/volumes" Sep 16 14:17:34.743099 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:34.743064 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" event={"ID":"535eb2f1-6571-45c3-97b4-aeac035271f4","Type":"ContainerStarted","Data":"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8"} Sep 16 14:17:34.760124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:34.760075 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" podStartSLOduration=2.248322606 podStartE2EDuration="2.760059908s" totalImagesPullingTime="511.737302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:34.758060476 +0000 UTC m=+585.335421629" watchObservedRunningTime="2026-09-16 14:17:34.760059908 +0000 UTC m=+585.337421101" Sep 16 14:17:39.945859 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:39.945824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:40.006843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.006812 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:40.031824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.031803 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.036728 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.036707 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:17:40.036979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.036962 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nc265\"" Sep 16 14:17:40.037769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.037752 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:17:40.037856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.037766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:17:40.053834 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.053807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:40.179415 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.179383 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:17:40.183238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.183215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.194845 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.194823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.194967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.194858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghqk\" (UniqueName: \"kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.194967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.194881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.203922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.203867 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:17:40.295974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.295937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.296134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.295997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7fq\" (UniqueName: \"kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.296134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.296021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.296134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.296045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghqk\" (UniqueName: \"kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.296134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.296074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.296134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.296099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.296304 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:40.296152 2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 16 14:17:40.296304 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:40.296222 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert podName:27ef3d09-7bab-4c44-8666-4c64fa18dc22 nodeName:}" failed. No retries permitted until 2026-09-16 14:17:40.796203299 +0000 UTC m=+591.373564429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert") pod "maas-controller-75dc6cbdf-z827f" (UID: "27ef3d09-7bab-4c44-8666-4c64fa18dc22") : secret "maas-controller-webhook-cert" not found Sep 16 14:17:40.298693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.298673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.313642 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.313621 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghqk\" (UniqueName: \"kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.397313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.397288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.397478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.397327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7fq\" (UniqueName: \"kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.397478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.397364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.399971 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.399945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.400071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.399948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.410976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.410950 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7fq\" (UniqueName: \"kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq\") pod \"maas-controller-77f9796c57-csmk6\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.492546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.492484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:40.587539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.587506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:40.694257 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.694219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:40.767113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.767090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:17:40.767537 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:40.767515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d4d9db_bd48_4142_a3ad_400612c8401f.slice/crio-d0616d9c26a3c9708a72edafa5f1e835fe76ec17ccebc230c636e22721466686 WatchSource:0}: Error finding container d0616d9c26a3c9708a72edafa5f1e835fe76ec17ccebc230c636e22721466686: Status 404 returned error can't find the container with id d0616d9c26a3c9708a72edafa5f1e835fe76ec17ccebc230c636e22721466686 Sep 16 14:17:40.777154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.777129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-csmk6" event={"ID":"94d4d9db-bd48-4142-a3ad-400612c8401f","Type":"ContainerStarted","Data":"d0616d9c26a3c9708a72edafa5f1e835fe76ec17ccebc230c636e22721466686"} Sep 16 14:17:40.800600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.800573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.802971 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.802955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") pod \"maas-controller-75dc6cbdf-z827f\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:40.942407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:40.942382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:41.219241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:41.219213 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:41.221340 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:17:41.221304 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ef3d09_7bab_4c44_8666_4c64fa18dc22.slice/crio-12ff9c3584994c9b7196c8d5f8dbf337c495c82ad565bc9ecc048e7416fbbbaf WatchSource:0}: Error finding container 12ff9c3584994c9b7196c8d5f8dbf337c495c82ad565bc9ecc048e7416fbbbaf: Status 404 returned error can't find the container with id 12ff9c3584994c9b7196c8d5f8dbf337c495c82ad565bc9ecc048e7416fbbbaf Sep 16 14:17:41.786550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:41.786435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-z827f" event={"ID":"27ef3d09-7bab-4c44-8666-4c64fa18dc22","Type":"ContainerStarted","Data":"12ff9c3584994c9b7196c8d5f8dbf337c495c82ad565bc9ecc048e7416fbbbaf"} Sep 16 14:17:43.616871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.616815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:43.724271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.724175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:43.801788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.801748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-csmk6" event={"ID":"94d4d9db-bd48-4142-a3ad-400612c8401f","Type":"ContainerStarted","Data":"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362"} Sep 16 14:17:43.802174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.801917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:43.803472 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.803445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-z827f" event={"ID":"27ef3d09-7bab-4c44-8666-4c64fa18dc22","Type":"ContainerStarted","Data":"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5"} Sep 16 14:17:43.803622 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.803535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:43.820535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.820464 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77f9796c57-csmk6" podStartSLOduration=1.154881136 podStartE2EDuration="3.820450904s" totalImagesPullingTime="2.665569768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:43.819261187 +0000 UTC m=+594.396622340" watchObservedRunningTime="2026-09-16 14:17:43.820450904 +0000 UTC m=+594.397812056" Sep 16 14:17:43.841498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:43.841449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-z827f" podStartSLOduration=2.6291760809999998 podStartE2EDuration="4.841432838s" totalImagesPullingTime="2.212256757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:43.838909891 +0000 UTC m=+594.416271044" watchObservedRunningTime="2026-09-16 14:17:43.841432838 +0000 UTC m=+594.418793989" Sep 16 14:17:49.798330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:49.798289 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:17:49.799429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:49.799408 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:17:54.810213 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:54.810183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:54.812655 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:54.810250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:17:54.880283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:54.880089 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:54.880518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:54.880325 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-z827f" podUID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" containerName="manager" containerID="cri-o://84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5" gracePeriod=30 Sep 16 14:17:55.129580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.129555 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:55.224061 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.224027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vghqk\" (UniqueName: \"kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk\") pod \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " Sep 16 14:17:55.224252 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.224089 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") pod \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " Sep 16 14:17:55.224252 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.224151 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs\") pod \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\" (UID: \"27ef3d09-7bab-4c44-8666-4c64fa18dc22\") " Sep 16 14:17:55.226523 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.226496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27ef3d09-7bab-4c44-8666-4c64fa18dc22" (UID: "27ef3d09-7bab-4c44-8666-4c64fa18dc22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:17:55.226853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.226836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk" (OuterVolumeSpecName: "kube-api-access-vghqk") pod "27ef3d09-7bab-4c44-8666-4c64fa18dc22" (UID: "27ef3d09-7bab-4c44-8666-4c64fa18dc22"). InnerVolumeSpecName "kube-api-access-vghqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:55.226918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.226846 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "27ef3d09-7bab-4c44-8666-4c64fa18dc22" (UID: "27ef3d09-7bab-4c44-8666-4c64fa18dc22"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:17:55.325575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.325541 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-webhook-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:55.325575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.325567 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef3d09-7bab-4c44-8666-4c64fa18dc22-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:55.325575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.325575 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghqk\" (UniqueName: \"kubernetes.io/projected/27ef3d09-7bab-4c44-8666-4c64fa18dc22-kube-api-access-vghqk\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:17:55.871370 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.871336 2579 generic.go:356] "Generic (PLEG): container finished" podID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" containerID="84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5" exitCode=0 Sep 16 14:17:55.871784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.871380 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-z827f" Sep 16 14:17:55.871784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.871418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-z827f" event={"ID":"27ef3d09-7bab-4c44-8666-4c64fa18dc22","Type":"ContainerDied","Data":"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5"} Sep 16 14:17:55.871784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.871466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-z827f" event={"ID":"27ef3d09-7bab-4c44-8666-4c64fa18dc22","Type":"ContainerDied","Data":"12ff9c3584994c9b7196c8d5f8dbf337c495c82ad565bc9ecc048e7416fbbbaf"} Sep 16 14:17:55.871784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.871487 2579 scope.go:117] "RemoveContainer" containerID="84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5" Sep 16 14:17:55.880583 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.880567 2579 scope.go:117] "RemoveContainer" containerID="84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5" Sep 16 14:17:55.880892 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:17:55.880873 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5\": container with ID starting with 84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5 not found: ID does not exist" containerID="84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5" Sep 16 14:17:55.880961 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.880903 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5"} err="failed to get container status \"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5\": rpc error: code = NotFound desc = could not find container \"84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5\": container with ID starting with 84d978ec4037c192653d7b902563ae16195881262d2cbc864140e73256e22ef5 not found: ID does not exist" Sep 16 14:17:55.891860 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.891837 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:55.898354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:55.898333 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-z827f"] Sep 16 14:17:57.817166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:17:57.817123 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" path="/var/lib/kubelet/pods/27ef3d09-7bab-4c44-8666-4c64fa18dc22/volumes" Sep 16 14:18:14.665938 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:14.665843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:16.307487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.307461 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r"] Sep 16 14:18:16.307904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.307887 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" containerName="manager" Sep 16 14:18:16.307971 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.307906 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" containerName="manager" Sep 16 14:18:16.308010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.307988 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ef3d09-7bab-4c44-8666-4c64fa18dc22" containerName="manager" Sep 16 14:18:16.314072 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.314052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.316912 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.316884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 14:18:16.316912 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.316906 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kj6xr\"" Sep 16 14:18:16.317908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.317894 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 14:18:16.334775 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.334750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r"] Sep 16 14:18:16.413991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.413961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzb4\" (UniqueName: \"kubernetes.io/projected/60017860-5440-4b4e-88ce-ddf6e09e182c-kube-api-access-qlzb4\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.414169 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.414002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-maas-api-tls\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.414169 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.414109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-metrics-certs\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.414283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.414167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60017860-5440-4b4e-88ce-ddf6e09e182c-openshift-service-ca\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.514594 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.514568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-maas-api-tls\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.514796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.514647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-metrics-certs\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.514796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.514677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60017860-5440-4b4e-88ce-ddf6e09e182c-openshift-service-ca\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.514796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.514702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzb4\" (UniqueName: \"kubernetes.io/projected/60017860-5440-4b4e-88ce-ddf6e09e182c-kube-api-access-qlzb4\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.515400 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.515375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60017860-5440-4b4e-88ce-ddf6e09e182c-openshift-service-ca\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.517466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.517438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-maas-api-tls\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.517632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.517587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60017860-5440-4b4e-88ce-ddf6e09e182c-metrics-certs\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.529273 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.529250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzb4\" (UniqueName: \"kubernetes.io/projected/60017860-5440-4b4e-88ce-ddf6e09e182c-kube-api-access-qlzb4\") pod \"maas-api-59f6566bbc-svr5r\" (UID: \"60017860-5440-4b4e-88ce-ddf6e09e182c\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.624450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.624367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:16.915386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.915356 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r"] Sep 16 14:18:16.916980 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:18:16.916956 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60017860_5440_4b4e_88ce_ddf6e09e182c.slice/crio-a91d07be4c6e88c372442d1cd27c2717b701ab7d1b1ba901532e975420be42ae WatchSource:0}: Error finding container a91d07be4c6e88c372442d1cd27c2717b701ab7d1b1ba901532e975420be42ae: Status 404 returned error can't find the container with id a91d07be4c6e88c372442d1cd27c2717b701ab7d1b1ba901532e975420be42ae Sep 16 14:18:16.918366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.918346 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:18:16.991975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:16.991942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" event={"ID":"60017860-5440-4b4e-88ce-ddf6e09e182c","Type":"ContainerStarted","Data":"a91d07be4c6e88c372442d1cd27c2717b701ab7d1b1ba901532e975420be42ae"} Sep 16 14:18:17.098733 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.098694 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:18:17.104063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.104040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.114466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.114446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 14:18:17.114745 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.114729 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mzsgb\"" Sep 16 14:18:17.130133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.130111 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:18:17.162587 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.162554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:17.220349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.220309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rnc\" (UniqueName: \"kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.220514 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.220357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.320969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.320917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rnc\" (UniqueName: \"kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.320969 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.320967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.321997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.321970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.341860 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.341832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rnc\" (UniqueName: \"kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc\") pod \"payload-pre-processing-59fbc47bf8-hw5sd\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.416239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.416198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:17.754196 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:17.754171 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:18:18.003017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:18.002980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" event={"ID":"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee","Type":"ContainerStarted","Data":"9aa77d924a95d02d8ec6bf2a614294c984d1c989d17345aace717a83e28f9508"} Sep 16 14:18:18.206864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:18.206823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:18.318348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:18.318304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:19.013276 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:19.013237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" event={"ID":"60017860-5440-4b4e-88ce-ddf6e09e182c","Type":"ContainerStarted","Data":"c2e6d2f23b5f439bd24382344f1480ad6acb6fa2359095331e5de959d66cdfeb"} Sep 16 14:18:19.013737 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:19.013396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:19.035349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:19.035302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" podStartSLOduration=1.630098867 podStartE2EDuration="3.03528569s" totalImagesPullingTime="1.405186823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:18:19.033791779 +0000 UTC m=+629.611152936" watchObservedRunningTime="2026-09-16 14:18:19.03528569 +0000 UTC m=+629.612646841" Sep 16 14:18:20.020097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:20.020054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" event={"ID":"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee","Type":"ContainerStarted","Data":"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75"} Sep 16 14:18:20.020531 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:20.020253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:20.043674 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:20.043623 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" podStartSLOduration=1.460959838 podStartE2EDuration="3.043591773s" totalImagesPullingTime="1.582631935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:18:20.041212418 +0000 UTC m=+630.618573570" watchObservedRunningTime="2026-09-16 14:18:20.043591773 +0000 UTC m=+630.620952924" Sep 16 14:18:25.025900 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:25.025872 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-svr5r" Sep 16 14:18:26.721737 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:26.721693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:26.858936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:26.858903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:30.445172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:30.445136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:31.028188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:31.027533 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:18:36.894259 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:36.894218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:42.373867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.373824 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6"] Sep 16 14:18:42.378649 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.378623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.382555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.382533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 14:18:42.382753 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.382731 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 14:18:42.382905 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.382560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vgn9b\"" Sep 16 14:18:42.383058 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.382677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 16 14:18:42.392359 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.392337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6"] Sep 16 14:18:42.408256 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.408208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:42.449305 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfwd\" (UniqueName: \"kubernetes.io/projected/8780e1f5-b584-45a1-bc40-7afc65c6fd83-kube-api-access-wjfwd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.449305 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.449491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.449491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.449491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.449491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.449445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.549933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.549900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfwd\" (UniqueName: \"kubernetes.io/projected/8780e1f5-b584-45a1-bc40-7afc65c6fd83-kube-api-access-wjfwd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.549947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.549996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.550651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.550550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.552326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.552305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8780e1f5-b584-45a1-bc40-7afc65c6fd83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.552651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.552593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8780e1f5-b584-45a1-bc40-7afc65c6fd83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.558679 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.558654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfwd\" (UniqueName: \"kubernetes.io/projected/8780e1f5-b584-45a1-bc40-7afc65c6fd83-kube-api-access-wjfwd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m6dj6\" (UID: \"8780e1f5-b584-45a1-bc40-7afc65c6fd83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.691763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.691732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:42.965840 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:42.965813 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6"] Sep 16 14:18:42.967670 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:18:42.967632 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8780e1f5_b584_45a1_bc40_7afc65c6fd83.slice/crio-ab8373aa362dc3c689d896cea20469e415f1153124a8767763442251ef5ce698 WatchSource:0}: Error finding container ab8373aa362dc3c689d896cea20469e415f1153124a8767763442251ef5ce698: Status 404 returned error can't find the container with id ab8373aa362dc3c689d896cea20469e415f1153124a8767763442251ef5ce698 Sep 16 14:18:43.156090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:43.156055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" event={"ID":"8780e1f5-b584-45a1-bc40-7afc65c6fd83","Type":"ContainerStarted","Data":"ab8373aa362dc3c689d896cea20469e415f1153124a8767763442251ef5ce698"} Sep 16 14:18:44.164741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:44.164702 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" event={"ID":"8780e1f5-b584-45a1-bc40-7afc65c6fd83","Type":"ContainerStarted","Data":"0d602054d5e67a7d2efa350795201b98a7ab28d20a8caba0110d137689adfdb1"} Sep 16 14:18:44.188241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:44.188188 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" podStartSLOduration=1.09387971 podStartE2EDuration="2.188172797s" totalImagesPullingTime="1.094293087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:18:44.185482578 +0000 UTC m=+654.762843730" watchObservedRunningTime="2026-09-16 14:18:44.188172797 +0000 UTC m=+654.765533948" Sep 16 14:18:52.691864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:52.691828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:52.692407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:52.691967 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:52.704519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:52.704496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:53.228585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:53.228554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m6dj6" Sep 16 14:18:55.834779 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:18:55.834742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:05.865336 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:05.865296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:09.136302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:09.136262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:19.319407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.319370 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk"] Sep 16 14:19:19.324859 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.324837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.329857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.329826 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 16 14:19:19.339595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.339571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk"] Sep 16 14:19:19.379087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:19.379510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.379653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4d534d-7005-4846-acc2-8f64f2b79d81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.379653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwstq\" (UniqueName: \"kubernetes.io/projected/9e4d534d-7005-4846-acc2-8f64f2b79d81-kube-api-access-qwstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.379653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.379809 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.379809 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.379729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480135 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480288 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4d534d-7005-4846-acc2-8f64f2b79d81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwstq\" (UniqueName: \"kubernetes.io/projected/9e4d534d-7005-4846-acc2-8f64f2b79d81-kube-api-access-qwstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480547 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480702 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.480778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.480763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.482544 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.482515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4d534d-7005-4846-acc2-8f64f2b79d81-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.482864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.482845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4d534d-7005-4846-acc2-8f64f2b79d81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.491739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.491720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwstq\" (UniqueName: \"kubernetes.io/projected/9e4d534d-7005-4846-acc2-8f64f2b79d81-kube-api-access-qwstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rrk\" (UID: \"9e4d534d-7005-4846-acc2-8f64f2b79d81\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.637047 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.636971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:19.916818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:19.916794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk"] Sep 16 14:19:19.918023 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:19:19.918000 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4d534d_7005_4846_acc2_8f64f2b79d81.slice/crio-b71d0992a7c5e484d1b79f046c656f4842c0cf644b6861eb154fa6db01784c42 WatchSource:0}: Error finding container b71d0992a7c5e484d1b79f046c656f4842c0cf644b6861eb154fa6db01784c42: Status 404 returned error can't find the container with id b71d0992a7c5e484d1b79f046c656f4842c0cf644b6861eb154fa6db01784c42 Sep 16 14:19:20.374770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:20.374734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" event={"ID":"9e4d534d-7005-4846-acc2-8f64f2b79d81","Type":"ContainerStarted","Data":"6b69ad2010dedf5d577df9fb0ecb48217b67c6b01e329df7e7d9b21639d85b20"} Sep 16 14:19:20.375173 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:20.374795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" event={"ID":"9e4d534d-7005-4846-acc2-8f64f2b79d81","Type":"ContainerStarted","Data":"b71d0992a7c5e484d1b79f046c656f4842c0cf644b6861eb154fa6db01784c42"} Sep 16 14:19:20.395847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:20.395799 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" podStartSLOduration=1.167330487 podStartE2EDuration="1.395782626s" totalImagesPullingTime="228.452139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:20.394467115 +0000 UTC m=+690.971828268" watchObservedRunningTime="2026-09-16 14:19:20.395782626 +0000 UTC m=+690.973143788" Sep 16 14:19:26.632661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:26.632599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:26.734942 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:26.734906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:28.190708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.190670 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg"] Sep 16 14:19:28.195383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.195362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.198489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.198410 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 16 14:19:28.203187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.203164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg"] Sep 16 14:19:28.254472 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.254671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.254671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wzp\" (UniqueName: \"kubernetes.io/projected/0f6f5288-18f7-4285-9eda-d7efa7e8db09-kube-api-access-j8wzp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.254671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.254671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.254849 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.254722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355588 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wzp\" (UniqueName: \"kubernetes.io/projected/0f6f5288-18f7-4285-9eda-d7efa7e8db09-kube-api-access-j8wzp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355866 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.355866 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.355818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.356130 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.356104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.356130 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.356123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.356241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.356140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.358664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.358638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6f5288-18f7-4285-9eda-d7efa7e8db09-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.358910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.358887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6f5288-18f7-4285-9eda-d7efa7e8db09-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.364955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.364933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wzp\" (UniqueName: \"kubernetes.io/projected/0f6f5288-18f7-4285-9eda-d7efa7e8db09-kube-api-access-j8wzp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg\" (UID: \"0f6f5288-18f7-4285-9eda-d7efa7e8db09\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.508883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.508799 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:28.729380 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.729340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:28.787067 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:28.787037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg"] Sep 16 14:19:28.788294 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:19:28.788265 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6f5288_18f7_4285_9eda_d7efa7e8db09.slice/crio-b5d66b5a12ead5586d19c1d7a202af97110e64c7182d053ac10962275288835b WatchSource:0}: Error finding container b5d66b5a12ead5586d19c1d7a202af97110e64c7182d053ac10962275288835b: Status 404 returned error can't find the container with id b5d66b5a12ead5586d19c1d7a202af97110e64c7182d053ac10962275288835b Sep 16 14:19:29.435389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.435353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" event={"ID":"0f6f5288-18f7-4285-9eda-d7efa7e8db09","Type":"ContainerStarted","Data":"5315b2ab475d919adc7869ed43580883423e73952f5b96d2c4f875da5c201530"} Sep 16 14:19:29.435809 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.435415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" event={"ID":"0f6f5288-18f7-4285-9eda-d7efa7e8db09","Type":"ContainerStarted","Data":"b5d66b5a12ead5586d19c1d7a202af97110e64c7182d053ac10962275288835b"} Sep 16 14:19:29.463931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.463885 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" podStartSLOduration=1.254451359 podStartE2EDuration="1.463869112s" totalImagesPullingTime="209.417753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:29.461291042 +0000 UTC m=+700.038652205" watchObservedRunningTime="2026-09-16 14:19:29.463869112 +0000 UTC m=+700.041230263" Sep 16 14:19:29.637718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.637683 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:29.637858 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.637810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:29.650948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:29.650928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:30.454146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:30.454113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rrk" Sep 16 14:19:32.611363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:32.611303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:36.813520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:36.813483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:38.359472 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:38.359428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:38.509687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:38.509657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:38.509890 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:38.509784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:38.522162 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:38.522143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:39.507404 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:39.507374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftc5fg" Sep 16 14:19:40.469102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:19:40.469067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:20:44.572107 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:20:44.572070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:20:44.670575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:20:44.670515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:01.252466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.252433 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:21:01.253076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.252675 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" podUID="535eb2f1-6571-45c3-97b4-aeac035271f4" containerName="authorino" containerID="cri-o://805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8" gracePeriod=30 Sep 16 14:21:01.508059 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.507991 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:21:01.588106 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.588075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r56r7\" (UniqueName: \"kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7\") pod \"535eb2f1-6571-45c3-97b4-aeac035271f4\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " Sep 16 14:21:01.588106 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.588113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert\") pod \"535eb2f1-6571-45c3-97b4-aeac035271f4\" (UID: \"535eb2f1-6571-45c3-97b4-aeac035271f4\") " Sep 16 14:21:01.590384 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.590356 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7" (OuterVolumeSpecName: "kube-api-access-r56r7") pod "535eb2f1-6571-45c3-97b4-aeac035271f4" (UID: "535eb2f1-6571-45c3-97b4-aeac035271f4"). InnerVolumeSpecName "kube-api-access-r56r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:21:01.602777 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.602747 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "535eb2f1-6571-45c3-97b4-aeac035271f4" (UID: "535eb2f1-6571-45c3-97b4-aeac035271f4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:01.664486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.664445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:01.689790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.689755 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56r7\" (UniqueName: \"kubernetes.io/projected/535eb2f1-6571-45c3-97b4-aeac035271f4-kube-api-access-r56r7\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:01.689790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.689783 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/535eb2f1-6571-45c3-97b4-aeac035271f4-tls-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:01.878333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.878304 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:21:01.878506 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.878485 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77f9796c57-csmk6" podUID="94d4d9db-bd48-4142-a3ad-400612c8401f" containerName="manager" containerID="cri-o://e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362" gracePeriod=30 Sep 16 14:21:01.952247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.952219 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:21:01.952444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.952420 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" podUID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" containerName="payload-pre-processing" containerID="cri-o://1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75" gracePeriod=30 Sep 16 14:21:01.989819 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:21:01.989786 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca470d1f_26f2_43d5_97ce_8c7b38b6fcee.slice/crio-conmon-1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:21:01.992564 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.992534 2579 generic.go:356] "Generic (PLEG): container finished" podID="535eb2f1-6571-45c3-97b4-aeac035271f4" containerID="805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8" exitCode=0 Sep 16 14:21:01.992692 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.992576 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" Sep 16 14:21:01.992692 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.992629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" event={"ID":"535eb2f1-6571-45c3-97b4-aeac035271f4","Type":"ContainerDied","Data":"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8"} Sep 16 14:21:01.992692 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.992673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bc75bc5b-65fq5" event={"ID":"535eb2f1-6571-45c3-97b4-aeac035271f4","Type":"ContainerDied","Data":"f24618a1dbd2ab7a32a162ec96057e08fd7e759656291675a8835191fd4b097b"} Sep 16 14:21:01.992692 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:01.992690 2579 scope.go:117] "RemoveContainer" containerID="805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8" Sep 16 14:21:02.028576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.028552 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:21:02.036117 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.036092 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55bc75bc5b-65fq5"] Sep 16 14:21:02.066365 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.066347 2579 scope.go:117] "RemoveContainer" containerID="805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8" Sep 16 14:21:02.066737 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:21:02.066717 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8\": container with ID starting with 805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8 not found: ID does not exist" containerID="805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8" Sep 16 14:21:02.066827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.066752 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8"} err="failed to get container status \"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8\": rpc error: code = NotFound desc = could not find container \"805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8\": container with ID starting with 805ff29817ca6c7ad70422f425291705edb953abf0d376c5a5fddd8fc5a407a8 not found: ID does not exist" Sep 16 14:21:02.207413 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.207386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:21:02.219768 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.219720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:02.238552 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.238532 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:21:02.294889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.294864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g7fq\" (UniqueName: \"kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq\") pod \"94d4d9db-bd48-4142-a3ad-400612c8401f\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " Sep 16 14:21:02.294889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.294900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert\") pod \"94d4d9db-bd48-4142-a3ad-400612c8401f\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " Sep 16 14:21:02.295365 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.294985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs\") pod \"94d4d9db-bd48-4142-a3ad-400612c8401f\" (UID: \"94d4d9db-bd48-4142-a3ad-400612c8401f\") " Sep 16 14:21:02.297677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.297646 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "94d4d9db-bd48-4142-a3ad-400612c8401f" (UID: "94d4d9db-bd48-4142-a3ad-400612c8401f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:02.297854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.297829 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94d4d9db-bd48-4142-a3ad-400612c8401f" (UID: "94d4d9db-bd48-4142-a3ad-400612c8401f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:02.298101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.298081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq" (OuterVolumeSpecName: "kube-api-access-7g7fq") pod "94d4d9db-bd48-4142-a3ad-400612c8401f" (UID: "94d4d9db-bd48-4142-a3ad-400612c8401f"). InnerVolumeSpecName "kube-api-access-7g7fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:21:02.337995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.337960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:02.396077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume\") pod \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " Sep 16 14:21:02.396077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396061 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rnc\" (UniqueName: \"kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc\") pod \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\" (UID: \"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee\") " Sep 16 14:21:02.396312 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396291 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7fq\" (UniqueName: \"kubernetes.io/projected/94d4d9db-bd48-4142-a3ad-400612c8401f-kube-api-access-7g7fq\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:02.396312 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396307 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-webhook-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:02.396428 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396316 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d4d9db-bd48-4142-a3ad-400612c8401f-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:02.396474 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.396415 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" (UID: "ca470d1f-26f2-43d5-97ce-8c7b38b6fcee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:21:02.398211 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.398184 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc" (OuterVolumeSpecName: "kube-api-access-79rnc") pod "ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" (UID: "ca470d1f-26f2-43d5-97ce-8c7b38b6fcee"). InnerVolumeSpecName "kube-api-access-79rnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:21:02.497145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.497115 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-plugins-config-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:02.497145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:02.497138 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rnc\" (UniqueName: \"kubernetes.io/projected/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee-kube-api-access-79rnc\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:21:03.000797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.000764 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" containerID="1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75" exitCode=0 Sep 16 14:21:03.000992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.000811 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" Sep 16 14:21:03.000992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.000843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" event={"ID":"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee","Type":"ContainerDied","Data":"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75"} Sep 16 14:21:03.000992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.000892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd" event={"ID":"ca470d1f-26f2-43d5-97ce-8c7b38b6fcee","Type":"ContainerDied","Data":"9aa77d924a95d02d8ec6bf2a614294c984d1c989d17345aace717a83e28f9508"} Sep 16 14:21:03.000992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.000914 2579 scope.go:117] "RemoveContainer" containerID="1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75" Sep 16 14:21:03.002520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.002484 2579 generic.go:356] "Generic (PLEG): container finished" podID="94d4d9db-bd48-4142-a3ad-400612c8401f" containerID="e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362" exitCode=0 Sep 16 14:21:03.002680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.002531 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-csmk6" Sep 16 14:21:03.002680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.002547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-csmk6" event={"ID":"94d4d9db-bd48-4142-a3ad-400612c8401f","Type":"ContainerDied","Data":"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362"} Sep 16 14:21:03.002680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.002629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-csmk6" event={"ID":"94d4d9db-bd48-4142-a3ad-400612c8401f","Type":"ContainerDied","Data":"d0616d9c26a3c9708a72edafa5f1e835fe76ec17ccebc230c636e22721466686"} Sep 16 14:21:03.010653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.010631 2579 scope.go:117] "RemoveContainer" containerID="1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75" Sep 16 14:21:03.010965 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:21:03.010948 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75\": container with ID starting with 1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75 not found: ID does not exist" containerID="1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75" Sep 16 14:21:03.011073 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.010977 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75"} err="failed to get container status \"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75\": rpc error: code = NotFound desc = could not find container \"1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75\": container with ID starting with 1d539e2db2c2edb8ca3c2fae68bf0aa56ce4fd35b987f92cb114ce335c1c4b75 not found: ID does not exist" Sep 16 14:21:03.011073 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.011000 2579 scope.go:117] "RemoveContainer" containerID="e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362" Sep 16 14:21:03.019423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.019399 2579 scope.go:117] "RemoveContainer" containerID="e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362" Sep 16 14:21:03.019733 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:21:03.019716 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362\": container with ID starting with e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362 not found: ID does not exist" containerID="e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362" Sep 16 14:21:03.019800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.019738 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362"} err="failed to get container status \"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362\": rpc error: code = NotFound desc = could not find container \"e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362\": container with ID starting with e3cf7734dfef658145dfc6e80d48924c59bbc9891309c29c10b65702818f5362 not found: ID does not exist" Sep 16 14:21:03.044627 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.042844 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:21:03.052059 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.052034 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77f9796c57-csmk6"] Sep 16 14:21:03.060647 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.060625 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:21:03.061037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061025 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" containerName="payload-pre-processing" Sep 16 14:21:03.061077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061039 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" containerName="payload-pre-processing" Sep 16 14:21:03.061077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061054 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535eb2f1-6571-45c3-97b4-aeac035271f4" containerName="authorino" Sep 16 14:21:03.061077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061059 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="535eb2f1-6571-45c3-97b4-aeac035271f4" containerName="authorino" Sep 16 14:21:03.061077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061072 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d4d9db-bd48-4142-a3ad-400612c8401f" containerName="manager" Sep 16 14:21:03.061077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061077 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d4d9db-bd48-4142-a3ad-400612c8401f" containerName="manager" Sep 16 14:21:03.061233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061143 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="535eb2f1-6571-45c3-97b4-aeac035271f4" containerName="authorino" Sep 16 14:21:03.061233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061152 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d4d9db-bd48-4142-a3ad-400612c8401f" containerName="manager" Sep 16 14:21:03.061233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.061158 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" containerName="payload-pre-processing" Sep 16 14:21:03.065521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.065505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.069379 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.069356 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:21:03.069499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.069420 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:21:03.069499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.069367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:21:03.069666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.069650 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nc265\"" Sep 16 14:21:03.077812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.077792 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:21:03.083546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.083520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:21:03.089049 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.089031 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hw5sd"] Sep 16 14:21:03.204589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.204560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.204749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.204597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qld\" (UniqueName: \"kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.204749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.204708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.305147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.305070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.305147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.305110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qld\" (UniqueName: \"kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.305536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.305247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.307713 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.307693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.307829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.307751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.314672 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.314645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qld\" (UniqueName: \"kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld\") pod \"maas-controller-77f9796c57-k7nb2\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.376500 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.376465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:03.669527 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.669503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:21:03.671126 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:21:03.671099 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf084cb_8674_4c38_a4ab_2b659d9df94a.slice/crio-da1ef06ba188ed097c53f6d2ffdd5e4599a839ee04cacb1a3fb43660046c1480 WatchSource:0}: Error finding container da1ef06ba188ed097c53f6d2ffdd5e4599a839ee04cacb1a3fb43660046c1480: Status 404 returned error can't find the container with id da1ef06ba188ed097c53f6d2ffdd5e4599a839ee04cacb1a3fb43660046c1480 Sep 16 14:21:03.817973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.817943 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535eb2f1-6571-45c3-97b4-aeac035271f4" path="/var/lib/kubelet/pods/535eb2f1-6571-45c3-97b4-aeac035271f4/volumes" Sep 16 14:21:03.818288 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.818275 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d4d9db-bd48-4142-a3ad-400612c8401f" path="/var/lib/kubelet/pods/94d4d9db-bd48-4142-a3ad-400612c8401f/volumes" Sep 16 14:21:03.818583 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:03.818572 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca470d1f-26f2-43d5-97ce-8c7b38b6fcee" path="/var/lib/kubelet/pods/ca470d1f-26f2-43d5-97ce-8c7b38b6fcee/volumes" Sep 16 14:21:04.012276 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:04.012244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-k7nb2" event={"ID":"5bf084cb-8674-4c38-a4ab-2b659d9df94a","Type":"ContainerStarted","Data":"da1ef06ba188ed097c53f6d2ffdd5e4599a839ee04cacb1a3fb43660046c1480"} Sep 16 14:21:05.021227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:05.021190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-k7nb2" event={"ID":"5bf084cb-8674-4c38-a4ab-2b659d9df94a","Type":"ContainerStarted","Data":"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884"} Sep 16 14:21:05.021644 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:05.021256 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:16.029801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:16.029723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:21:16.056833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:16.056779 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77f9796c57-k7nb2" podStartSLOduration=12.323405683 podStartE2EDuration="13.056764883s" totalImagesPullingTime="733.3592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:05.046651743 +0000 UTC m=+795.624012892" watchObservedRunningTime="2026-09-16 14:21:16.056764883 +0000 UTC m=+806.634126034" Sep 16 14:21:21.260885 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.260851 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d"] Sep 16 14:21:21.264570 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.264546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.273690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.273670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 14:21:21.273807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.273670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mzsgb\"" Sep 16 14:21:21.282482 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.282457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:21.295147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.295117 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d"] Sep 16 14:21:21.360567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.360538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f13b6-39b3-4135-9a21-3dee9312165f-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.360567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.360573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9glq\" (UniqueName: \"kubernetes.io/projected/6b6f13b6-39b3-4135-9a21-3dee9312165f-kube-api-access-k9glq\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.430129 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.430099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:21.461303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.461269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f13b6-39b3-4135-9a21-3dee9312165f-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.461453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.461311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9glq\" (UniqueName: \"kubernetes.io/projected/6b6f13b6-39b3-4135-9a21-3dee9312165f-kube-api-access-k9glq\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.462116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.462098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f13b6-39b3-4135-9a21-3dee9312165f-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.472348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.472322 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9glq\" (UniqueName: \"kubernetes.io/projected/6b6f13b6-39b3-4135-9a21-3dee9312165f-kube-api-access-k9glq\") pod \"payload-pre-processing-6cd864bc95-5mm9d\" (UID: \"6b6f13b6-39b3-4135-9a21-3dee9312165f\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.574106 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.574008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:21.868197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:21.868172 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d"] Sep 16 14:21:21.869909 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:21:21.869883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f13b6_39b3_4135_9a21_3dee9312165f.slice/crio-7b6897c5857af3295215f32c8f7682b6ab01c7f8924785b89814de73741d6444 WatchSource:0}: Error finding container 7b6897c5857af3295215f32c8f7682b6ab01c7f8924785b89814de73741d6444: Status 404 returned error can't find the container with id 7b6897c5857af3295215f32c8f7682b6ab01c7f8924785b89814de73741d6444 Sep 16 14:21:22.121096 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:22.121007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" event={"ID":"6b6f13b6-39b3-4135-9a21-3dee9312165f","Type":"ContainerStarted","Data":"7b6897c5857af3295215f32c8f7682b6ab01c7f8924785b89814de73741d6444"} Sep 16 14:21:25.141533 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:25.141498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" event={"ID":"6b6f13b6-39b3-4135-9a21-3dee9312165f","Type":"ContainerStarted","Data":"f3ba18b4983b7858f9a375f5b79efbf01a2cb758a1776f3e76cd3428c0eee097"} Sep 16 14:21:25.141998 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:25.141653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:25.141998 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:25.141737 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:21:25.168170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:25.168120 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" podStartSLOduration=1.618890135 podStartE2EDuration="4.168101912s" totalImagesPullingTime="2.549211777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:25.166220549 +0000 UTC m=+815.743581702" watchObservedRunningTime="2026-09-16 14:21:25.168101912 +0000 UTC m=+815.745463064" Sep 16 14:21:26.147653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:26.147625 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:21:36.149814 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:36.149777 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" Sep 16 14:21:36.150309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:36.150150 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:21:47.224322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:47.224292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:47.546910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:47.546823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:48.752440 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:48.752403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:48.858283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:48.858252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:50.578448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:50.578420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:50.797074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:21:50.797030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:07.469590 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:07.469556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:07.573242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:07.573205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:07.897628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:07.897525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:08.039743 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.039694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:08.080889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.080861 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:22:08.086918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.086896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.090855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.090837 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c37e6e-openshift-default-dockercfg-n6s55\"" Sep 16 14:22:08.093372 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.093350 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 14:22:08.104566 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.104543 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:22:08.155280 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.155167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:08.163114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdkm\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.163481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.163414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.263990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdkm\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.264965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.265007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.264971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.265304 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.265284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.266756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.266730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.266986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.266965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.288889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.288856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdkm\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.289048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.289024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token\") pod \"e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.397040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.397006 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:08.492504 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.492452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:08.582144 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.582103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:08.685172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:08.685144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:22:08.687034 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:22:08.687003 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf620444_55c6_4031_81bc_716e8b742c24.slice/crio-100cfd9b9a07f8208eac1c39290b8542a888dfa4ea7472a8043d346bae9e3947 WatchSource:0}: Error finding container 100cfd9b9a07f8208eac1c39290b8542a888dfa4ea7472a8043d346bae9e3947: Status 404 returned error can't find the container with id 100cfd9b9a07f8208eac1c39290b8542a888dfa4ea7472a8043d346bae9e3947 Sep 16 14:22:09.393231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:09.393194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerStarted","Data":"100cfd9b9a07f8208eac1c39290b8542a888dfa4ea7472a8043d346bae9e3947"} Sep 16 14:22:11.078408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:11.078378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:11.335029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:11.334919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:11.598002 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:11.597876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:12.186461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.186427 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:12.186752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.186500 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:12.186752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.186530 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:12.280394 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.280361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:22:12.284774 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.284755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.304806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.304774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.305026 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.305004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.305172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.305154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28m8m\" (UniqueName: \"kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.305318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.305303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.309015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.308995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d48266-serving-cert\"" Sep 16 14:22:12.331502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.330309 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:22:12.406052 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.406019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.406052 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.406056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28m8m\" (UniqueName: \"kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.406294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.406083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.406294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.406193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.406859 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.406830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.408793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.408768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.408953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.408939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.417206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.417176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerStarted","Data":"d2d01543122ce0988e07f00f1c8edb728053519e03b17245bde56d1fd533ae56"} Sep 16 14:22:12.431381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.431348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28m8m\" (UniqueName: \"kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m\") pod \"maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.449220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.449155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:12.488858 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.488811 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podStartSLOduration=0.991836787 podStartE2EDuration="4.488795271s" totalImagesPullingTime="3.496958484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:12.484674874 +0000 UTC m=+863.062036025" watchObservedRunningTime="2026-09-16 14:22:12.488795271 +0000 UTC m=+863.066156474" Sep 16 14:22:12.597576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.597546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:12.718000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.717920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:12.912206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:12.912180 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:22:12.912550 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:22:12.912533 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464ea653_88af_409c_9b5b_57f5f008f689.slice/crio-06063acbf1d71fdfe1c476dfcc7d59147a31ae69cd61899f61fdade219d43473 WatchSource:0}: Error finding container 06063acbf1d71fdfe1c476dfcc7d59147a31ae69cd61899f61fdade219d43473: Status 404 returned error can't find the container with id 06063acbf1d71fdfe1c476dfcc7d59147a31ae69cd61899f61fdade219d43473 Sep 16 14:22:13.397529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.397505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:13.402704 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.402682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:13.424837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.424815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" event={"ID":"464ea653-88af-409c-9b5b-57f5f008f689","Type":"ContainerStarted","Data":"06063acbf1d71fdfe1c476dfcc7d59147a31ae69cd61899f61fdade219d43473"} Sep 16 14:22:13.425063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.425049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:13.425987 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.425970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:13.512396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.512367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:13.690337 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:13.690298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:14.215654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.215624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:22:14.219623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.219591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.223133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.223116 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-11636c-serving-cert\"" Sep 16 14:22:14.238802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.238777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:22:14.324026 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.323998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.324192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.324039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.324192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.324122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.324275 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.324199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpws\" (UniqueName: \"kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.386364 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.386326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:14.425580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.425542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.426045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.425595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.426045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.425696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.426045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.425763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpws\" (UniqueName: \"kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.426045 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:22:14.425836 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-serving-cert: secret "maas-api-e2e-worker-w0-11636c-serving-cert" not found Sep 16 14:22:14.426045 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:22:14.425909 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls podName:ef293318-eeb1-4b97-871d-91db6db018ef nodeName:}" failed. No retries permitted until 2026-09-16 14:22:14.925888509 +0000 UTC m=+865.503249643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls") pod "maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" (UID: "ef293318-eeb1-4b97-871d-91db6db018ef") : secret "maas-api-e2e-worker-w0-11636c-serving-cert" not found Sep 16 14:22:14.426379 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.426357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.428316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.428289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.434146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.434114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" event={"ID":"464ea653-88af-409c-9b5b-57f5f008f689","Type":"ContainerStarted","Data":"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8"} Sep 16 14:22:14.434291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.434275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:14.443032 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.443011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpws\" (UniqueName: \"kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.470573 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.470477 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" podStartSLOduration=2.024744198 podStartE2EDuration="2.470461541s" totalImagesPullingTime="445.717343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:14.470125887 +0000 UTC m=+865.047487050" watchObservedRunningTime="2026-09-16 14:22:14.470461541 +0000 UTC m=+865.047822713" Sep 16 14:22:14.503755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.503724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:14.803372 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.803284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:14.932100 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.932056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:14.934818 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:14.934781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") pod \"maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:15.083319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:15.083231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:15.132182 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:15.132151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:15.343821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:15.343710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:15.436296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:15.436267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:22:15.438209 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:22:15.438182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef293318_eeb1_4b97_871d_91db6db018ef.slice/crio-08b49651d4f3fe25c85b7dd29d55cb5641cba3bd065044ce3ff1b14fab83e38e WatchSource:0}: Error finding container 08b49651d4f3fe25c85b7dd29d55cb5641cba3bd065044ce3ff1b14fab83e38e: Status 404 returned error can't find the container with id 08b49651d4f3fe25c85b7dd29d55cb5641cba3bd065044ce3ff1b14fab83e38e Sep 16 14:22:15.565146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:15.565102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:16.448911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.448875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" event={"ID":"ef293318-eeb1-4b97-871d-91db6db018ef","Type":"ContainerStarted","Data":"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c"} Sep 16 14:22:16.449295 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.448967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" event={"ID":"ef293318-eeb1-4b97-871d-91db6db018ef","Type":"ContainerStarted","Data":"08b49651d4f3fe25c85b7dd29d55cb5641cba3bd065044ce3ff1b14fab83e38e"} Sep 16 14:22:16.449295 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.449126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:16.524122 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.524074 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" podStartSLOduration=2.190354426 podStartE2EDuration="2.524058332s" totalImagesPullingTime="333.703906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:16.518873102 +0000 UTC m=+867.096234254" watchObservedRunningTime="2026-09-16 14:22:16.524058332 +0000 UTC m=+867.101419483" Sep 16 14:22:16.554285 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.554248 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:22:16.558101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.558084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.567316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.567293 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c37e6e-serving-cert\"" Sep 16 14:22:16.576347 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.576323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:22:16.577847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.577813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:16.647712 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.647676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.647878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.647749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztf4\" (UniqueName: \"kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.647878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.647836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.647946 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.647914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.749332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.749254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztf4\" (UniqueName: \"kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.749332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.749306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.749539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.749386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.749539 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.749422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.750149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.750070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.751872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.751848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.752081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.752061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.775024 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.775003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztf4\" (UniqueName: \"kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4\") pod \"maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:16.845942 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.845907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:16.869813 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:16.869782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:17.169924 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:22:17.169898 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7a0af3_f0f7_465a_a9ea_85640218ca2f.slice/crio-bc43eaf91fe53c227c5f58961ca231e72902cded6bd45cf8c34d72339481b741 WatchSource:0}: Error finding container bc43eaf91fe53c227c5f58961ca231e72902cded6bd45cf8c34d72339481b741: Status 404 returned error can't find the container with id bc43eaf91fe53c227c5f58961ca231e72902cded6bd45cf8c34d72339481b741 Sep 16 14:22:17.170226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:17.170207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:22:17.459471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:17.459436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" event={"ID":"ec7a0af3-f0f7-465a-a9ea-85640218ca2f","Type":"ContainerStarted","Data":"bc43eaf91fe53c227c5f58961ca231e72902cded6bd45cf8c34d72339481b741"} Sep 16 14:22:17.620307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:17.620282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:17.773540 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:17.773447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:18.395889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:18.395850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:18.467479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:18.467446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" event={"ID":"ec7a0af3-f0f7-465a-a9ea-85640218ca2f","Type":"ContainerStarted","Data":"1cd7a6fcebc887f9001e4643349f28add88a650bb227338d06b53543d8e5a053"} Sep 16 14:22:18.467879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:18.467516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:18.501185 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:18.501139 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" podStartSLOduration=2.12319314 podStartE2EDuration="2.501124407s" totalImagesPullingTime="377.931267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:18.499687802 +0000 UTC m=+869.077048950" watchObservedRunningTime="2026-09-16 14:22:18.501124407 +0000 UTC m=+869.078485559" Sep 16 14:22:19.612305 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:19.612268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:20.446918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:20.446888 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:22:20.667181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:20.667147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:22.465246 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:22.465221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:22:24.479441 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:24.479414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:22:25.537948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:25.537915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:25.778466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:25.778425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:25.967847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:25.967809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:29.465831 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:29.465785 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 14:22:29.466333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:29.465931 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 14:22:32.563653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.563599 2579 generic.go:356] "Generic (PLEG): container finished" podID="df620444-55c6-4031-81bc-716e8b742c24" containerID="d2d01543122ce0988e07f00f1c8edb728053519e03b17245bde56d1fd533ae56" exitCode=0 Sep 16 14:22:32.564081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.563675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerDied","Data":"d2d01543122ce0988e07f00f1c8edb728053519e03b17245bde56d1fd533ae56"} Sep 16 14:22:32.564159 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.564141 2579 scope.go:117] "RemoveContainer" containerID="d2d01543122ce0988e07f00f1c8edb728053519e03b17245bde56d1fd533ae56" Sep 16 14:22:32.565241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.565206 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:32.565347 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.565296 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:32.565347 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:32.565342 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:22:33.453661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:33.453597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:33.571876 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:33.571845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerStarted","Data":"3a8084485f322197cef43e3a8f8f9504393df9769a9f7a65d560a8c9fc2f9b83"} Sep 16 14:22:33.619898 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:33.619865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:33.800375 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:33.800265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:34.026659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:34.026589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:34.297691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:34.297653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:34.397945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:34.397913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:34.398233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:34.398215 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 14:22:34.398303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:34.398269 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 14:22:35.398089 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:35.398053 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 14:22:35.398498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:35.398106 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 14:22:36.069627 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.069583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:36.167251 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.167203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:36.398258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.398173 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 14:22:36.398258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.398230 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 14:22:36.737358 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.737317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:36.891420 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:36.891388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:37.397808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:37.397766 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 14:22:37.398004 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:37.397838 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 14:22:37.570796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:37.570760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:38.398339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:38.398297 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 14:22:38.398592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:38.398361 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 14:22:39.143041 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.143005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:39.401060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.400978 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:39.401401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.401388 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:39.402184 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.402166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:22:39.675022 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.674991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:39.981240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:39.981141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:40.272798 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:40.272708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:40.753623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:40.753578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:41.195076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.195040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:41.526255 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.526223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:41.669921 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.669888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:41.811228 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.811144 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:22:41.840662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.840624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:41.962549 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:41.962505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:43.456497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:43.456465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:43.564977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:43.564940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:44.267641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:44.267586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:44.417513 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:44.417483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:44.950878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:44.950842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:46.254355 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:46.254321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:47.516131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:47.516093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:48.713919 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.713882 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:22:48.717691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.717674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.721408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.721385 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0adaea-serving-cert\"" Sep 16 14:22:48.753018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.752993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:48.767087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.767062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:22:48.790618 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.790578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.790764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.790633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5trd\" (UniqueName: \"kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.790764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.790663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.790894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.790776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.892077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.892039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5trd\" (UniqueName: \"kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.892077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.892077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.892367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.892263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.892499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.892479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.893101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.893074 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.894966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.894949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.895040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.894998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.904212 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.904190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5trd\" (UniqueName: \"kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd\") pod \"maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:48.974008 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:48.973914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:49.028510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.028473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:49.319371 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.319294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:22:49.327519 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:22:49.327490 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28b5bc4_57fb_4428_9b7a_614a331c780d.slice/crio-89fd269f6fbd3899ffcc3aea98fb18dcca36db000b5c75056178ee0fb4bda2c4 WatchSource:0}: Error finding container 89fd269f6fbd3899ffcc3aea98fb18dcca36db000b5c75056178ee0fb4bda2c4: Status 404 returned error can't find the container with id 89fd269f6fbd3899ffcc3aea98fb18dcca36db000b5c75056178ee0fb4bda2c4 Sep 16 14:22:49.673984 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.673953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" event={"ID":"c28b5bc4-57fb-4428-9b7a-614a331c780d","Type":"ContainerStarted","Data":"89fd269f6fbd3899ffcc3aea98fb18dcca36db000b5c75056178ee0fb4bda2c4"} Sep 16 14:22:49.691211 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.691161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:49.831224 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.831192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:49.850201 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.850173 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:22:49.850460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.850438 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:22:49.986455 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:49.986364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:50.681675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:50.681635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" event={"ID":"c28b5bc4-57fb-4428-9b7a-614a331c780d","Type":"ContainerStarted","Data":"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d"} Sep 16 14:22:50.681871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:50.681794 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:22:50.729701 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:50.729642 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" podStartSLOduration=2.304948025 podStartE2EDuration="2.729620559s" totalImagesPullingTime="424.672534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:50.726381766 +0000 UTC m=+901.303742932" watchObservedRunningTime="2026-09-16 14:22:50.729620559 +0000 UTC m=+901.306981703" Sep 16 14:22:56.693894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:22:56.693855 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:23:06.787807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:06.787766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:13.513313 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:13.513272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:14.278793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:14.278746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:14.691398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:14.691368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:16.326212 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:16.326173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:17.482948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:17.482906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:28.600364 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:28.600326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:23:53.085123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:23:53.085083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:24:05.810986 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:05.810952 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:24:14.573727 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.573651 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:24:14.577541 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.577521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.585676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.585655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-11636c\"/\"kube-root-ca.crt\"" Sep 16 14:24:14.585788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.585661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11636c\"/\"premium-w0-11636c-kserve-self-signed-certs\"" Sep 16 14:24:14.585788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.585689 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11636c\"/\"default-dockercfg-rnmfp\"" Sep 16 14:24:14.585788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.585700 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-11636c\"/\"openshift-service-ca.crt\"" Sep 16 14:24:14.602879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.602857 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:24:14.629450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.629419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:24:14.711383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.711519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.711519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.711519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.711519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.711696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.711572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.812928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.813014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.812985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.815055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.815032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.815297 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.815282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.822585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.822563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz\") pod \"premium-w0-11636c-kserve-6c749d8cc8-k8djv\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:14.886405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:14.886332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:15.171951 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:15.171872 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:24:15.175417 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:24:15.175388 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod527f6558_dcde_4c68_9c32_1abcddf90f9b.slice/crio-8224667ba9b6fc99a1ee981217350c81751cfdbee2c71a9a9d7995d0aed8f678 WatchSource:0}: Error finding container 8224667ba9b6fc99a1ee981217350c81751cfdbee2c71a9a9d7995d0aed8f678: Status 404 returned error can't find the container with id 8224667ba9b6fc99a1ee981217350c81751cfdbee2c71a9a9d7995d0aed8f678 Sep 16 14:24:15.194430 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:15.194399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" event={"ID":"527f6558-dcde-4c68-9c32-1abcddf90f9b","Type":"ContainerStarted","Data":"8224667ba9b6fc99a1ee981217350c81751cfdbee2c71a9a9d7995d0aed8f678"} Sep 16 14:24:16.202989 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:16.202951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" event={"ID":"527f6558-dcde-4c68-9c32-1abcddf90f9b","Type":"ContainerStarted","Data":"737698b2aa17bd3d5425b0b4aed7db163a734fd954f4a79d8740f7f48ec613b6"} Sep 16 14:24:16.238204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:16.238153 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" podStartSLOduration=2.238138197 podStartE2EDuration="2.238138197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:24:16.238046391 +0000 UTC m=+986.815407544" watchObservedRunningTime="2026-09-16 14:24:16.238138197 +0000 UTC m=+986.815499349" Sep 16 14:24:24.886959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:24.886917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:24.887502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:24.887214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:24.900144 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:24.900118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:25.270207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:25.270175 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:24:44.705239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:24:44.705206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:25:07.794668 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:07.794629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:25:14.315495 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.315458 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8948-bdcd97"] Sep 16 14:25:14.322194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.322170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:14.328516 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.328488 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8948-bdcd97"] Sep 16 14:25:14.329939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.329915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kdfll\"" Sep 16 14:25:14.403093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.403061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5z5\" (UniqueName: \"kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5\") pod \"test-curl-8948-bdcd97\" (UID: \"cbee4f5a-23f7-4199-a7b7-86b26a17b546\") " pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:14.504411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.504379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5z5\" (UniqueName: \"kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5\") pod \"test-curl-8948-bdcd97\" (UID: \"cbee4f5a-23f7-4199-a7b7-86b26a17b546\") " pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:14.518554 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.518518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5z5\" (UniqueName: \"kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5\") pod \"test-curl-8948-bdcd97\" (UID: \"cbee4f5a-23f7-4199-a7b7-86b26a17b546\") " pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:14.634194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.634109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:14.908077 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.908039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8948-bdcd97"] Sep 16 14:25:14.909380 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:25:14.909352 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbee4f5a_23f7_4199_a7b7_86b26a17b546.slice/crio-3a39de43f24002758b689521bb86e11811ad5ae62147dca14dbfef1b5fc5b00b WatchSource:0}: Error finding container 3a39de43f24002758b689521bb86e11811ad5ae62147dca14dbfef1b5fc5b00b: Status 404 returned error can't find the container with id 3a39de43f24002758b689521bb86e11811ad5ae62147dca14dbfef1b5fc5b00b Sep 16 14:25:14.911364 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:14.911349 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:25:15.572456 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:15.572422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8948-bdcd97" event={"ID":"cbee4f5a-23f7-4199-a7b7-86b26a17b546","Type":"ContainerStarted","Data":"3a39de43f24002758b689521bb86e11811ad5ae62147dca14dbfef1b5fc5b00b"} Sep 16 14:25:17.588254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:17.588217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8948-bdcd97" event={"ID":"cbee4f5a-23f7-4199-a7b7-86b26a17b546","Type":"ContainerStarted","Data":"225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a"} Sep 16 14:25:17.604797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:17.604754 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8948-bdcd97" podStartSLOduration=1.735733787 podStartE2EDuration="3.604739253s" totalImagesPullingTime="1.869005466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:25:17.603882243 +0000 UTC m=+1048.181243396" watchObservedRunningTime="2026-09-16 14:25:17.604739253 +0000 UTC m=+1048.182100449" Sep 16 14:25:18.281301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.281198 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8948-bdcd97"] Sep 16 14:25:18.284589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.284562 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8948-bdcd97"] Sep 16 14:25:18.420291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.420255 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8948-491749"] Sep 16 14:25:18.420771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.420755 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" containerName="test-curl-8948-bdcd97" Sep 16 14:25:18.420821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.420773 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" containerName="test-curl-8948-bdcd97" Sep 16 14:25:18.420857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.420851 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" containerName="test-curl-8948-bdcd97" Sep 16 14:25:18.425810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.425781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:18.435932 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.435906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8948-491749"] Sep 16 14:25:18.546156 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.546063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2vb\" (UniqueName: \"kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb\") pod \"test-curl-8948-491749\" (UID: \"ad15da62-1cf8-47fc-af44-69d0f5fef54b\") " pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:18.594346 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:18.594309 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8948-bdcd97" container="test-curl-8948-bdcd97" Sep 16 14:25:18.647062 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.647033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2vb\" (UniqueName: \"kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb\") pod \"test-curl-8948-491749\" (UID: \"ad15da62-1cf8-47fc-af44-69d0f5fef54b\") " pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:18.659200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.659169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2vb\" (UniqueName: \"kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb\") pod \"test-curl-8948-491749\" (UID: \"ad15da62-1cf8-47fc-af44-69d0f5fef54b\") " pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:18.737523 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:18.737491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:19.014480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:19.014450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8948-491749"] Sep 16 14:25:19.015529 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:25:19.015500 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad15da62_1cf8_47fc_af44_69d0f5fef54b.slice/crio-6ba908cc4110b2d794739fa00a65c1242533fecfe89dac5a8b94d17db8e8192e WatchSource:0}: Error finding container 6ba908cc4110b2d794739fa00a65c1242533fecfe89dac5a8b94d17db8e8192e: Status 404 returned error can't find the container with id 6ba908cc4110b2d794739fa00a65c1242533fecfe89dac5a8b94d17db8e8192e Sep 16 14:25:19.602011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:19.601966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8948-491749" event={"ID":"ad15da62-1cf8-47fc-af44-69d0f5fef54b","Type":"ContainerStarted","Data":"f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf"} Sep 16 14:25:19.602408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:19.602019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8948-491749" event={"ID":"ad15da62-1cf8-47fc-af44-69d0f5fef54b","Type":"ContainerStarted","Data":"6ba908cc4110b2d794739fa00a65c1242533fecfe89dac5a8b94d17db8e8192e"} Sep 16 14:25:19.602408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:19.602122 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8948-bdcd97" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" containerName="test-curl-8948-bdcd97" containerID="cri-o://225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a" gracePeriod=2 Sep 16 14:25:19.619222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:19.619180 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8948-491749" podStartSLOduration=1.619165605 podStartE2EDuration="1.619165605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:25:19.618856664 +0000 UTC m=+1050.196217819" watchObservedRunningTime="2026-09-16 14:25:19.619165605 +0000 UTC m=+1050.196526757" Sep 16 14:25:19.621523 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:19.621495 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8948-bdcd97\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:19.818415 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:19.818376 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8948-bdcd97\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:20.315737 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:20.315630 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8948-491749"] Sep 16 14:25:20.319913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:20.319886 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8948-491749"] Sep 16 14:25:20.608509 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:20.608428 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8948-491749" container="test-curl-8948-491749" Sep 16 14:25:21.614907 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:21.614875 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8948-491749" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" containerName="test-curl-8948-491749" containerID="cri-o://f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf" gracePeriod=2 Sep 16 14:25:21.617535 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:21.617497 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8948-491749\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:21.760405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:21.760382 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:21.763231 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:21.763204 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8948-491749\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:21.879388 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:21.879309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn5z5\" (UniqueName: \"kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5\") pod \"cbee4f5a-23f7-4199-a7b7-86b26a17b546\" (UID: \"cbee4f5a-23f7-4199-a7b7-86b26a17b546\") " Sep 16 14:25:21.881510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:21.881487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5" (OuterVolumeSpecName: "kube-api-access-gn5z5") pod "cbee4f5a-23f7-4199-a7b7-86b26a17b546" (UID: "cbee4f5a-23f7-4199-a7b7-86b26a17b546"). InnerVolumeSpecName "kube-api-access-gn5z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:25:21.980414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:21.980382 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5z5\" (UniqueName: \"kubernetes.io/projected/cbee4f5a-23f7-4199-a7b7-86b26a17b546-kube-api-access-gn5z5\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:25:22.622667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:22.622635 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" containerID="225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a" exitCode=137 Sep 16 14:25:22.623092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:22.622676 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-bdcd97" Sep 16 14:25:22.623092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:22.622727 2579 scope.go:117] "RemoveContainer" containerID="225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a" Sep 16 14:25:22.633091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:22.633070 2579 scope.go:117] "RemoveContainer" containerID="225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a" Sep 16 14:25:22.633395 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:22.633373 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a\": container with ID starting with 225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a not found: ID does not exist" containerID="225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a" Sep 16 14:25:22.633487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:22.633403 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a"} err="failed to get container status \"225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a\": rpc error: code = NotFound desc = could not find container \"225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a\": container with ID starting with 225769aa76e319a78685c6735bb3f3bc63ea8bed3b8ad515ba1342ef1984b71a not found: ID does not exist" Sep 16 14:25:23.817859 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:23.817819 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbee4f5a-23f7-4199-a7b7-86b26a17b546" path="/var/lib/kubelet/pods/cbee4f5a-23f7-4199-a7b7-86b26a17b546/volumes" Sep 16 14:25:24.203096 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.203071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:24.303329 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.303295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk2vb\" (UniqueName: \"kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb\") pod \"ad15da62-1cf8-47fc-af44-69d0f5fef54b\" (UID: \"ad15da62-1cf8-47fc-af44-69d0f5fef54b\") " Sep 16 14:25:24.305498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.305472 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb" (OuterVolumeSpecName: "kube-api-access-kk2vb") pod "ad15da62-1cf8-47fc-af44-69d0f5fef54b" (UID: "ad15da62-1cf8-47fc-af44-69d0f5fef54b"). InnerVolumeSpecName "kube-api-access-kk2vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:25:24.405173 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.405090 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2vb\" (UniqueName: \"kubernetes.io/projected/ad15da62-1cf8-47fc-af44-69d0f5fef54b-kube-api-access-kk2vb\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:25:24.639665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.639632 2579 generic.go:356] "Generic (PLEG): container finished" podID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" containerID="f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf" exitCode=137 Sep 16 14:25:24.639853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.639699 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:24.639853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.639745 2579 scope.go:117] "RemoveContainer" containerID="f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf" Sep 16 14:25:24.649595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.649574 2579 scope.go:117] "RemoveContainer" containerID="f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf" Sep 16 14:25:24.649955 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:24.649927 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf\": container with ID starting with f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf not found: ID does not exist" containerID="f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf" Sep 16 14:25:24.650031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.649965 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf"} err="failed to get container status \"f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf\": rpc error: code = NotFound desc = could not find container \"f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf\": container with ID starting with f7db77523a64e107a6648506b018fbee9f928e1e371601af633f987b869e5baf not found: ID does not exist" Sep 16 14:25:24.657746 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:25:24.657682 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8948-491749\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" pod="opendatahub/test-curl-8948-491749" Sep 16 14:25:24.811623 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:24.811587 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:25:25.819153 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:25.819118 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" path="/var/lib/kubelet/pods/ad15da62-1cf8-47fc-af44-69d0f5fef54b/volumes" Sep 16 14:25:43.940865 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:25:43.940773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:05.531200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:05.531158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:08.527843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:08.527803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:08.629064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:08.629017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:08.976669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:08.976631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:12.103546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:12.103513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:16.010349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:16.010317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:16.078341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:16.078306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:16.239033 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:16.238994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:16.546847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:16.546818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:16.960316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:16.960279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:18.962745 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:18.962712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:19.639551 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:19.639515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:19.917109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:19.917009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:20.139956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:20.139920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:21.146952 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:21.146917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:21.946148 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:21.946111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:22.801700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.801666 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:26:22.802476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.802453 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" containerName="test-curl-8948-491749" Sep 16 14:26:22.802553 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.802480 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" containerName="test-curl-8948-491749" Sep 16 14:26:22.802678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.802661 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad15da62-1cf8-47fc-af44-69d0f5fef54b" containerName="test-curl-8948-491749" Sep 16 14:26:22.806003 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.805982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.808922 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.808899 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-286f09d7-serving-cert\"" Sep 16 14:26:22.816974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.816954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:26:22.839463 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.839431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:22.889570 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.889532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8whn5\" (UniqueName: \"kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.889791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.889589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.889791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.889658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.889791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.889686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.939399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.939356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:22.990931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.990901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.991147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.990941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.991147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.990963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.991147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.991054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8whn5\" (UniqueName: \"kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.991572 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.991548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.993731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.993703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:22.993825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:22.993779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:23.000934 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.000907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8whn5\" (UniqueName: \"kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5\") pod \"maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:23.039747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.039703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:23.119815 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.119736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:23.317169 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.317135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:23.401856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.401826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:26:23.403213 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:26:23.403122 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad53cbc_1e9c_4b6f_bd5a_65f21032d24e.slice/crio-041240788ba0b352b4bce8ed1b23b0fdd1ea88faba02d2c4e45025fcb3c5b893 WatchSource:0}: Error finding container 041240788ba0b352b4bce8ed1b23b0fdd1ea88faba02d2c4e45025fcb3c5b893: Status 404 returned error can't find the container with id 041240788ba0b352b4bce8ed1b23b0fdd1ea88faba02d2c4e45025fcb3c5b893 Sep 16 14:26:23.403397 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.403347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:23.838331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:23.838300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:24.014983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:24.014889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" event={"ID":"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e","Type":"ContainerStarted","Data":"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef"} Sep 16 14:26:24.014983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:24.014948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" event={"ID":"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e","Type":"ContainerStarted","Data":"041240788ba0b352b4bce8ed1b23b0fdd1ea88faba02d2c4e45025fcb3c5b893"} Sep 16 14:26:24.014983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:24.014973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:24.039812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:24.039760 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" podStartSLOduration=1.680597456 podStartE2EDuration="2.03974327s" totalImagesPullingTime="359.145814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:26:24.036076092 +0000 UTC m=+1114.613437243" watchObservedRunningTime="2026-09-16 14:26:24.03974327 +0000 UTC m=+1114.617104421" Sep 16 14:26:24.742941 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:24.742908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:30.029276 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:30.029245 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:26:42.811860 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:42.811822 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:26:43.296571 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:43.296530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:44.763622 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:44.763570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:45.774348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:45.774308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:46.771368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:46.771334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:48.148538 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:48.148502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:49.309355 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:49.309324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:49.535880 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:49.535843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:50.120239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:50.120205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:50.743016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:50.742983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:52.558839 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:52.558800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:53.110137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:53.110094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:54.184005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:54.183962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:55.082423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.082381 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:26:55.082689 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.082660 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" podUID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" containerName="maas-api" containerID="cri-o://1cd7a6fcebc887f9001e4643349f28add88a650bb227338d06b53543d8e5a053" gracePeriod=30 Sep 16 14:26:55.145822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.145788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:55.218567 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.218538 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" containerID="1cd7a6fcebc887f9001e4643349f28add88a650bb227338d06b53543d8e5a053" exitCode=0 Sep 16 14:26:55.219182 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.218663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" event={"ID":"ec7a0af3-f0f7-465a-a9ea-85640218ca2f","Type":"ContainerDied","Data":"1cd7a6fcebc887f9001e4643349f28add88a650bb227338d06b53543d8e5a053"} Sep 16 14:26:55.344444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.344369 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:26:55.356732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.356707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:55.500580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.500545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:55.509323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls\") pod \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " Sep 16 14:26:55.509424 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca\") pod \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " Sep 16 14:26:55.509424 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509407 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ztf4\" (UniqueName: \"kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4\") pod \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " Sep 16 14:26:55.509548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs\") pod \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\" (UID: \"ec7a0af3-f0f7-465a-a9ea-85640218ca2f\") " Sep 16 14:26:55.509808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509779 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec7a0af3-f0f7-465a-a9ea-85640218ca2f" (UID: "ec7a0af3-f0f7-465a-a9ea-85640218ca2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:26:55.509924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.509885 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:26:55.512093 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.512065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec7a0af3-f0f7-465a-a9ea-85640218ca2f" (UID: "ec7a0af3-f0f7-465a-a9ea-85640218ca2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:26:55.512216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.512198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec7a0af3-f0f7-465a-a9ea-85640218ca2f" (UID: "ec7a0af3-f0f7-465a-a9ea-85640218ca2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:26:55.512281 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.512228 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4" (OuterVolumeSpecName: "kube-api-access-6ztf4") pod "ec7a0af3-f0f7-465a-a9ea-85640218ca2f" (UID: "ec7a0af3-f0f7-465a-a9ea-85640218ca2f"). InnerVolumeSpecName "kube-api-access-6ztf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:26:55.611104 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.611020 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:26:55.611104 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.611048 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztf4\" (UniqueName: \"kubernetes.io/projected/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-kube-api-access-6ztf4\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:26:55.611104 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.611056 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7a0af3-f0f7-465a-a9ea-85640218ca2f-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:26:55.668933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.668895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:55.829266 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:55.829232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:56.057758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.057716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:56.226549 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.226519 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" Sep 16 14:26:56.227054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.226554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz" event={"ID":"ec7a0af3-f0f7-465a-a9ea-85640218ca2f","Type":"ContainerDied","Data":"bc43eaf91fe53c227c5f58961ca231e72902cded6bd45cf8c34d72339481b741"} Sep 16 14:26:56.227054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.226691 2579 scope.go:117] "RemoveContainer" containerID="1cd7a6fcebc887f9001e4643349f28add88a650bb227338d06b53543d8e5a053" Sep 16 14:26:56.262156 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.262129 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:26:56.266180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:56.266156 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c37e6e-8549cbb7b-qdlhz"] Sep 16 14:26:57.021416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:57.021373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:57.817990 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:57.817951 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" path="/var/lib/kubelet/pods/ec7a0af3-f0f7-465a-a9ea-85640218ca2f/volumes" Sep 16 14:26:58.335587 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:58.335552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:26:59.475301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:26:59.475264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:00.565403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:00.565362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:01.504311 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.504280 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:27:01.504656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.504620 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" podUID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" containerName="maas-api" containerID="cri-o://2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef" gracePeriod=30 Sep 16 14:27:01.591306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.591274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:01.785991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.785963 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:27:01.790797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.790750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:01.865783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.865752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca\") pod \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " Sep 16 14:27:01.865783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.865788 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls\") pod \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " Sep 16 14:27:01.866031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.865864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs\") pod \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " Sep 16 14:27:01.866031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.865913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8whn5\" (UniqueName: \"kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5\") pod \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\" (UID: \"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e\") " Sep 16 14:27:01.866175 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.866149 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" (UID: "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:27:01.867009 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.866985 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:01.868290 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.868259 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" (UID: "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:01.868887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.868859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" (UID: "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:01.868887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.868879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5" (OuterVolumeSpecName: "kube-api-access-8whn5") pod "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" (UID: "cad53cbc-1e9c-4b6f-bd5a-65f21032d24e"). InnerVolumeSpecName "kube-api-access-8whn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:01.905663 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.905636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:01.968192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.968159 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:01.968192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.968185 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8whn5\" (UniqueName: \"kubernetes.io/projected/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-kube-api-access-8whn5\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:01.968192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:01.968196 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:02.268996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.268960 2579 generic.go:356] "Generic (PLEG): container finished" podID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" containerID="2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef" exitCode=0 Sep 16 14:27:02.269181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.269004 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" Sep 16 14:27:02.269181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.269008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" event={"ID":"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e","Type":"ContainerDied","Data":"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef"} Sep 16 14:27:02.269181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.269127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz" event={"ID":"cad53cbc-1e9c-4b6f-bd5a-65f21032d24e","Type":"ContainerDied","Data":"041240788ba0b352b4bce8ed1b23b0fdd1ea88faba02d2c4e45025fcb3c5b893"} Sep 16 14:27:02.269181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.269152 2579 scope.go:117] "RemoveContainer" containerID="2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef" Sep 16 14:27:02.280741 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.280712 2579 scope.go:117] "RemoveContainer" containerID="2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef" Sep 16 14:27:02.281044 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:02.281023 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef\": container with ID starting with 2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef not found: ID does not exist" containerID="2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef" Sep 16 14:27:02.281107 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.281054 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef"} err="failed to get container status \"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef\": rpc error: code = NotFound desc = could not find container \"2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef\": container with ID starting with 2ba03620defb9e8b0f62c9b88d85cde65521139cbc476be8098cd889c1ed0bef not found: ID does not exist" Sep 16 14:27:02.312625 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.310363 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:27:02.322509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.322484 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-286f09d7-5585bd85b6-z9srz"] Sep 16 14:27:02.362822 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.362777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:02.536346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.536262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:02.751643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:02.751584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:03.198344 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.198306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:03.276911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.276884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:03.277362 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277348 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" containerName="maas-api" Sep 16 14:27:03.277362 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" containerName="maas-api" Sep 16 14:27:03.277509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277373 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" containerName="maas-api" Sep 16 14:27:03.277509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277381 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" containerName="maas-api" Sep 16 14:27:03.277653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277555 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7a0af3-f0f7-465a-a9ea-85640218ca2f" containerName="maas-api" Sep 16 14:27:03.277750 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.277714 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" containerName="maas-api" Sep 16 14:27:03.282931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.282914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.286522 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.286504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-beaf471d-gw-openshift-default-dockercfg-49d6j\"" Sep 16 14:27:03.316665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.316638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:03.379934 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.379905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.379934 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.379938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr88x\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380236 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380236 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380236 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.380332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.380231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.390750 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.390726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:03.481075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr88x\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481744 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.481793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.481780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.482128 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.482023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.483796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.483774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.483943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.483925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.515486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.515458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.521714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.521691 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr88x\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x\") pod \"e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.592804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.592769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:03.819343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.819263 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad53cbc-1e9c-4b6f-bd5a-65f21032d24e" path="/var/lib/kubelet/pods/cad53cbc-1e9c-4b6f-bd5a-65f21032d24e/volumes" Sep 16 14:27:03.897630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.897585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:03.898249 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:27:03.898225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c169a9_29a5_423f_a391_f2d328001baa.slice/crio-03f267bfdc848a11a4a09e24f9e22f9c0e7ef66ee0f6f64f8fe14a2282abe4a9 WatchSource:0}: Error finding container 03f267bfdc848a11a4a09e24f9e22f9c0e7ef66ee0f6f64f8fe14a2282abe4a9: Status 404 returned error can't find the container with id 03f267bfdc848a11a4a09e24f9e22f9c0e7ef66ee0f6f64f8fe14a2282abe4a9 Sep 16 14:27:03.903336 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.903297 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:03.903437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.903370 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:03.903437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:03.903403 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:04.063163 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.063117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:04.285707 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.285675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerStarted","Data":"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444"} Sep 16 14:27:04.285875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.285735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerStarted","Data":"03f267bfdc848a11a4a09e24f9e22f9c0e7ef66ee0f6f64f8fe14a2282abe4a9"} Sep 16 14:27:04.319107 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.319075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:04.334376 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.334320 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podStartSLOduration=1.334298783 podStartE2EDuration="1.334298783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:04.331148348 +0000 UTC m=+1154.908509501" watchObservedRunningTime="2026-09-16 14:27:04.334298783 +0000 UTC m=+1154.911659937" Sep 16 14:27:04.593155 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.593074 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:04.598530 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:04.598503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:05.292890 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:05.292855 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:05.293726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:05.293710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:05.602166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:05.602075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:06.308046 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:06.308005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:06.469778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:06.469738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:07.526668 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:07.526562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:07.864870 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:07.864777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:07.970599 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:07.970569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:08.140043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.139959 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:27:08.140267 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.140242 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" containerID="cri-o://3a8084485f322197cef43e3a8f8f9504393df9769a9f7a65d560a8c9fc2f9b83" gracePeriod=30 Sep 16 14:27:08.177827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.177797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:08.410207 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.410126 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:08.414269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.414240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.418445 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.418424 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-beaf471d-serving-cert\"" Sep 16 14:27:08.430851 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.430810 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:08.465600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.465563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:08.536479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.536446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8x9\" (UniqueName: \"kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.536920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.536582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.536920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.536641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.536920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.536715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.568424 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.568371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:08.637939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.637898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.638103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.637948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.638103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.637990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.638220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.638115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8x9\" (UniqueName: \"kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.638867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.638835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.641300 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.641282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.641437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.641416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.653258 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.653238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8x9\" (UniqueName: \"kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9\") pod \"maas-api-e2e-migrate-beaf471d-66685995d9-wh65g\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:08.688355 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.688321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:08.725709 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:08.725682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:09.018459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.018426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:09.020214 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:27:09.020192 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8199b4_ad10_4e8f_a75f_210cf2c07d1b.slice/crio-77e520e22e5dec29d027b21c621ebeff0171aa1912da35339b51ead3be6e10a3 WatchSource:0}: Error finding container 77e520e22e5dec29d027b21c621ebeff0171aa1912da35339b51ead3be6e10a3: Status 404 returned error can't find the container with id 77e520e22e5dec29d027b21c621ebeff0171aa1912da35339b51ead3be6e10a3 Sep 16 14:27:09.317021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.316933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" event={"ID":"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b","Type":"ContainerStarted","Data":"77e520e22e5dec29d027b21c621ebeff0171aa1912da35339b51ead3be6e10a3"} Sep 16 14:27:09.403646 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.403595 2579 patch_prober.go:28] interesting pod/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 14:27:09.403767 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.403669 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 14:27:09.566168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.566129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:09.723061 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:09.723032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:10.324654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:10.324615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" event={"ID":"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b","Type":"ContainerStarted","Data":"3f664f3f80f7a153826de2fd7ab4e995ff0d6abdda01a241cdfd5806d559674d"} Sep 16 14:27:10.324858 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:10.324686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:10.363511 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:10.363459 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" podStartSLOduration=1.994963631 podStartE2EDuration="2.363443983s" totalImagesPullingTime="368.480352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:10.362058685 +0000 UTC m=+1160.939419838" watchObservedRunningTime="2026-09-16 14:27:10.363443983 +0000 UTC m=+1160.940805131" Sep 16 14:27:10.532549 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:10.532516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:11.332975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:11.332916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:11.705090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:11.705051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:12.620721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:12.620684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:13.354159 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.354119 2579 generic.go:356] "Generic (PLEG): container finished" podID="df620444-55c6-4031-81bc-716e8b742c24" containerID="3a8084485f322197cef43e3a8f8f9504393df9769a9f7a65d560a8c9fc2f9b83" exitCode=0 Sep 16 14:27:13.354333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.354228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerDied","Data":"3a8084485f322197cef43e3a8f8f9504393df9769a9f7a65d560a8c9fc2f9b83"} Sep 16 14:27:13.354333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.354287 2579 scope.go:117] "RemoveContainer" containerID="d2d01543122ce0988e07f00f1c8edb728053519e03b17245bde56d1fd533ae56" Sep 16 14:27:13.416634 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.416552 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:27:13.490689 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490657 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.490868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490711 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.490868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.490868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.490868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490814 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.490868 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.491121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.491121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490928 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqdkm\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.491121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket\") pod \"df620444-55c6-4031-81bc-716e8b742c24\" (UID: \"df620444-55c6-4031-81bc-716e8b742c24\") " Sep 16 14:27:13.491121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.490987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data" (OuterVolumeSpecName: "istio-data") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:13.491121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491082 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:13.491343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491273 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:13.491343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:27:13.491343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491332 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-data\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.491456 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491348 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.491456 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.491427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:13.493252 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.493222 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:13.493486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.493456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:27:13.493486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.493454 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token" (OuterVolumeSpecName: "istio-token") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:13.493634 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.493526 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm" (OuterVolumeSpecName: "kube-api-access-mqdkm") pod "df620444-55c6-4031-81bc-716e8b742c24" (UID: "df620444-55c6-4031-81bc-716e8b742c24"). InnerVolumeSpecName "kube-api-access-mqdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:13.579377 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.579338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:13.592521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592493 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df620444-55c6-4031-81bc-716e8b742c24-istio-podinfo\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592518 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-istio-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592527 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-credential-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592536 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdkm\" (UniqueName: \"kubernetes.io/projected/df620444-55c6-4031-81bc-716e8b742c24-kube-api-access-mqdkm\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592546 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-workload-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592555 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df620444-55c6-4031-81bc-716e8b742c24-istio-envoy\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:13.592739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:13.592563 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df620444-55c6-4031-81bc-716e8b742c24-istiod-ca-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:14.361429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:14.361397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" event={"ID":"df620444-55c6-4031-81bc-716e8b742c24","Type":"ContainerDied","Data":"100cfd9b9a07f8208eac1c39290b8542a888dfa4ea7472a8043d346bae9e3947"} Sep 16 14:27:14.361938 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:14.361458 2579 scope.go:117] "RemoveContainer" containerID="3a8084485f322197cef43e3a8f8f9504393df9769a9f7a65d560a8c9fc2f9b83" Sep 16 14:27:14.361938 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:14.361409 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7" Sep 16 14:27:14.403943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:14.403913 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:27:14.409726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:14.409701 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c37e6e-openshift-default-65455df964-g6qf7"] Sep 16 14:27:15.818136 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:15.818107 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df620444-55c6-4031-81bc-716e8b742c24" path="/var/lib/kubelet/pods/df620444-55c6-4031-81bc-716e8b742c24/volumes" Sep 16 14:27:16.336875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:16.336845 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:18.683471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:18.683433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:19.554948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.554919 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:27:19.555141 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.555120 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" podUID="c28b5bc4-57fb-4428-9b7a-614a331c780d" containerName="maas-api" containerID="cri-o://16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d" gracePeriod=30 Sep 16 14:27:19.636204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.636171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:19.833436 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.833411 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:27:19.922854 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.922794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:19.956556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.956521 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca\") pod \"c28b5bc4-57fb-4428-9b7a-614a331c780d\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " Sep 16 14:27:19.956749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.956572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5trd\" (UniqueName: \"kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd\") pod \"c28b5bc4-57fb-4428-9b7a-614a331c780d\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " Sep 16 14:27:19.956749 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.956741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs\") pod \"c28b5bc4-57fb-4428-9b7a-614a331c780d\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " Sep 16 14:27:19.956867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.956766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls\") pod \"c28b5bc4-57fb-4428-9b7a-614a331c780d\" (UID: \"c28b5bc4-57fb-4428-9b7a-614a331c780d\") " Sep 16 14:27:19.956924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.956906 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c28b5bc4-57fb-4428-9b7a-614a331c780d" (UID: "c28b5bc4-57fb-4428-9b7a-614a331c780d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:27:19.957137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.957119 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28b5bc4-57fb-4428-9b7a-614a331c780d-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:19.958793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.958767 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd" (OuterVolumeSpecName: "kube-api-access-f5trd") pod "c28b5bc4-57fb-4428-9b7a-614a331c780d" (UID: "c28b5bc4-57fb-4428-9b7a-614a331c780d"). InnerVolumeSpecName "kube-api-access-f5trd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:19.958920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.958896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c28b5bc4-57fb-4428-9b7a-614a331c780d" (UID: "c28b5bc4-57fb-4428-9b7a-614a331c780d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:19.958962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:19.958942 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c28b5bc4-57fb-4428-9b7a-614a331c780d" (UID: "c28b5bc4-57fb-4428-9b7a-614a331c780d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:20.058641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.058578 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5trd\" (UniqueName: \"kubernetes.io/projected/c28b5bc4-57fb-4428-9b7a-614a331c780d-kube-api-access-f5trd\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:20.058641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.058621 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:20.058641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.058638 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28b5bc4-57fb-4428-9b7a-614a331c780d-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:20.076380 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.076292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:20.167903 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.167869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:20.402487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.402393 2579 generic.go:356] "Generic (PLEG): container finished" podID="c28b5bc4-57fb-4428-9b7a-614a331c780d" containerID="16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d" exitCode=0 Sep 16 14:27:20.402487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.402447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" Sep 16 14:27:20.402742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.402487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" event={"ID":"c28b5bc4-57fb-4428-9b7a-614a331c780d","Type":"ContainerDied","Data":"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d"} Sep 16 14:27:20.402742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.402541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8" event={"ID":"c28b5bc4-57fb-4428-9b7a-614a331c780d","Type":"ContainerDied","Data":"89fd269f6fbd3899ffcc3aea98fb18dcca36db000b5c75056178ee0fb4bda2c4"} Sep 16 14:27:20.402742 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.402562 2579 scope.go:117] "RemoveContainer" containerID="16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d" Sep 16 14:27:20.413411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.413395 2579 scope.go:117] "RemoveContainer" containerID="16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d" Sep 16 14:27:20.413700 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:20.413677 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d\": container with ID starting with 16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d not found: ID does not exist" containerID="16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d" Sep 16 14:27:20.413763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.413713 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d"} err="failed to get container status \"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d\": rpc error: code = NotFound desc = could not find container \"16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d\": container with ID starting with 16499d634dbeeb83ebc620ae701c86909f5b71d0c4f3bd85190fcbabb94ec45d not found: ID does not exist" Sep 16 14:27:20.437161 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.437138 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:27:20.441728 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.441706 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adaea-59b8d7dbd6-7tnz8"] Sep 16 14:27:20.563372 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.563338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:20.892226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:20.892197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:21.556499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:21.556465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:21.818246 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:21.818162 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c28b5bc4-57fb-4428-9b7a-614a331c780d" path="/var/lib/kubelet/pods/c28b5bc4-57fb-4428-9b7a-614a331c780d/volumes" Sep 16 14:27:23.495933 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:23.495901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:24.009493 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:24.009463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:26.855631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:26.855579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:28.375125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:28.375088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:29.763473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:29.763436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:30.048875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:30.048838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:30.932262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:30.932228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:31.234508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.234423 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:27:31.234909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.234895 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28b5bc4-57fb-4428-9b7a-614a331c780d" containerName="maas-api" Sep 16 14:27:31.234958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.234911 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28b5bc4-57fb-4428-9b7a-614a331c780d" containerName="maas-api" Sep 16 14:27:31.234958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.234927 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:31.234958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.234933 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:31.235057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.235033 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:31.235057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.235041 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c28b5bc4-57fb-4428-9b7a-614a331c780d" containerName="maas-api" Sep 16 14:27:31.239714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.239692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.249479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.249455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c0974e-openshift-default-dockercfg-n8jl2\"" Sep 16 14:27:31.275143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.275116 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:27:31.293989 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.293961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:31.372021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.371991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpw6\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.372509 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.372403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.473884 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.473848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.473896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.473918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.473969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpw6\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474583 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474686 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474765 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.474815 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.474755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.477073 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.477050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.477278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.477260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.483080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.483056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:31.484222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.484202 2579 generic.go:356] "Generic (PLEG): container finished" podID="09c169a9-29a5-423f-a391-f2d328001baa" containerID="8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444" exitCode=0 Sep 16 14:27:31.484318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.484279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerDied","Data":"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444"} Sep 16 14:27:31.484763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.484716 2579 scope.go:117] "RemoveContainer" containerID="8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444" Sep 16 14:27:31.485912 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.485883 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:31.486001 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.485947 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:31.486001 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.485978 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:31.502545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.502526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.502806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.502788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpw6\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6\") pod \"e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.551805 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.551778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:31.829292 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.829203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:31.874050 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.874018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:27:31.875900 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:27:31.875872 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d19d5d2_8acc_4d54_8ad7_5c72708d1690.slice/crio-9580c71c05d0afac7a56df41678b21780c32df58cc2f767e0994b46aa578e4f0 WatchSource:0}: Error finding container 9580c71c05d0afac7a56df41678b21780c32df58cc2f767e0994b46aa578e4f0: Status 404 returned error can't find the container with id 9580c71c05d0afac7a56df41678b21780c32df58cc2f767e0994b46aa578e4f0 Sep 16 14:27:31.878122 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.878088 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:31.878206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.878156 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:31.878206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:31.878191 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:27:32.331564 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.331528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:32.493170 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.493138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerStarted","Data":"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12"} Sep 16 14:27:32.493353 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.493185 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerStarted","Data":"9580c71c05d0afac7a56df41678b21780c32df58cc2f767e0994b46aa578e4f0"} Sep 16 14:27:32.495109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.495082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerStarted","Data":"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a"} Sep 16 14:27:32.546046 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.545963 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podStartSLOduration=1.5459372249999999 podStartE2EDuration="1.545937225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:32.541397422 +0000 UTC m=+1183.118758575" watchObservedRunningTime="2026-09-16 14:27:32.545937225 +0000 UTC m=+1183.123298380" Sep 16 14:27:32.552696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.552662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:32.554034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.553999 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:27:32.554188 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.554059 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:27:32.593469 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.593387 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:32.594620 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.594585 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:32.594686 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:32.594651 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:33.561007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:33.560976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:33.593705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:33.593675 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:33.593847 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:33.593724 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:33.751827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:33.751723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:34.508244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.508215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:34.509429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.509404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:27:34.593823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.593787 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:34.594181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.593841 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:34.715260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.715226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:34.964456 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:34.964414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:35.292799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:35.292713 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:35.593299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:35.593212 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:35.593299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:35.593275 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:36.043580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:36.043547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:36.593490 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:36.593455 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:36.593694 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:36.593509 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:37.313880 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:37.313832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:37.593459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:37.593371 2579 patch_prober.go:28] interesting pod/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 16 14:27:37.593459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:37.593426 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 16 14:27:38.271711 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.271674 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:38.271906 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.271871 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" podUID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" containerName="maas-api" containerID="cri-o://3f664f3f80f7a153826de2fd7ab4e995ff0d6abdda01a241cdfd5806d559674d" gracePeriod=30 Sep 16 14:27:38.413593 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.413540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:38.542102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.542066 2579 generic.go:356] "Generic (PLEG): container finished" podID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" containerID="3f664f3f80f7a153826de2fd7ab4e995ff0d6abdda01a241cdfd5806d559674d" exitCode=0 Sep 16 14:27:38.542244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.542163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" event={"ID":"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b","Type":"ContainerDied","Data":"3f664f3f80f7a153826de2fd7ab4e995ff0d6abdda01a241cdfd5806d559674d"} Sep 16 14:27:38.557155 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.557129 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:38.597408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.597376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:38.598481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.598461 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:38.648872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.648842 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs\") pod \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " Sep 16 14:27:38.649025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.648946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw8x9\" (UniqueName: \"kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9\") pod \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " Sep 16 14:27:38.649025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.648967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls\") pod \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " Sep 16 14:27:38.649025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.648995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca\") pod \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\" (UID: \"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b\") " Sep 16 14:27:38.649512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.649395 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" (UID: "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:27:38.651342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.651318 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" (UID: "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:38.651470 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.651342 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" (UID: "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:27:38.651683 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.651666 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9" (OuterVolumeSpecName: "kube-api-access-lw8x9") pod "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" (UID: "2e8199b4-ad10-4e8f-a75f-210cf2c07d1b"). InnerVolumeSpecName "kube-api-access-lw8x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:38.750837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.750756 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8x9\" (UniqueName: \"kubernetes.io/projected/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-kube-api-access-lw8x9\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:38.750837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.750789 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:38.750837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.750804 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:38.750837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.750817 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:38.814410 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:38.814320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:39.040592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.040505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:39.207291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.207252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:39.317300 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.317260 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:39.416759 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.416725 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:27:39.417368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417236 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:39.417368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417258 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:39.417368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417299 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" containerName="maas-api" Sep 16 14:27:39.417368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417308 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" containerName="maas-api" Sep 16 14:27:39.417703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417444 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df620444-55c6-4031-81bc-716e8b742c24" containerName="istio-proxy" Sep 16 14:27:39.417703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.417464 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" containerName="maas-api" Sep 16 14:27:39.421132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.421113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.424592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.424576 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c0974e-serving-cert\"" Sep 16 14:27:39.440150 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.440116 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:27:39.440414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.440383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:39.456469 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.456427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg927\" (UniqueName: \"kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.456691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.456494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.456691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.456569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.456801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.456725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.550174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.550078 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" Sep 16 14:27:39.550174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.550105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g" event={"ID":"2e8199b4-ad10-4e8f-a75f-210cf2c07d1b","Type":"ContainerDied","Data":"77e520e22e5dec29d027b21c621ebeff0171aa1912da35339b51ead3be6e10a3"} Sep 16 14:27:39.550174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.550169 2579 scope.go:117] "RemoveContainer" containerID="3f664f3f80f7a153826de2fd7ab4e995ff0d6abdda01a241cdfd5806d559674d" Sep 16 14:27:39.550812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.550759 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" secret="" err="secret \"e2e-migrate-beaf471d-gw-openshift-default-dockercfg-49d6j\" not found" Sep 16 14:27:39.557393 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.557363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.557529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.557460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg927\" (UniqueName: \"kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.557529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.557517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.557676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.557562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.558223 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.558191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.560961 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.560937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.561441 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.561421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.572197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.572168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg927\" (UniqueName: \"kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927\") pod \"maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.609102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.609075 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:39.619479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.619454 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-beaf471d-66685995d9-wh65g"] Sep 16 14:27:39.732092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.732055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:39.821510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:39.821438 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8199b4-ad10-4e8f-a75f-210cf2c07d1b" path="/var/lib/kubelet/pods/2e8199b4-ad10-4e8f-a75f-210cf2c07d1b/volumes" Sep 16 14:27:40.058744 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.058723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:27:40.061499 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:27:40.061470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd266b49f_3ab4_4296_9bec_cf3920316e17.slice/crio-6a1cff134cbd741fe19f94a60c824fc4955592eac932b0988fd1295a792d2360 WatchSource:0}: Error finding container 6a1cff134cbd741fe19f94a60c824fc4955592eac932b0988fd1295a792d2360: Status 404 returned error can't find the container with id 6a1cff134cbd741fe19f94a60c824fc4955592eac932b0988fd1295a792d2360 Sep 16 14:27:40.316935 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.316897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:40.462266 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.462232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:40.559980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.559951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:40.560338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.560310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" event={"ID":"d266b49f-3ab4-4296-9bec-cf3920316e17","Type":"ContainerStarted","Data":"6a1cff134cbd741fe19f94a60c824fc4955592eac932b0988fd1295a792d2360"} Sep 16 14:27:40.561621 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.561575 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" containerID="cri-o://d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a" gracePeriod=30 Sep 16 14:27:40.663433 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:40.663397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:41.028701 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:41.028666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:41.571005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:41.570965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" event={"ID":"d266b49f-3ab4-4296-9bec-cf3920316e17","Type":"ContainerStarted","Data":"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e"} Sep 16 14:27:41.571386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:41.571030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:41.591831 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:41.591774 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" podStartSLOduration=2.131991294 podStartE2EDuration="2.591757549s" totalImagesPullingTime="459.766255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:41.588404451 +0000 UTC m=+1192.165765618" watchObservedRunningTime="2026-09-16 14:27:41.591757549 +0000 UTC m=+1192.169118700" Sep 16 14:27:42.157959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:42.157926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:43.018695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:43.018663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:43.128354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:43.128318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:43.275238 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:43.275152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:43.403448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:43.403418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:44.310654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.310596 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:27:44.314554 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.314531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.318763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.318716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5f53f6be-serving-cert\"" Sep 16 14:27:44.330401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.330376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:27:44.404562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.404519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.404721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.404580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.404780 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.404714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgf6\" (UniqueName: \"kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.404815 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.404780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.410653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.410623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:44.505864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.505830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgf6\" (UniqueName: \"kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.506051 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.505895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.506051 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.505947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.506051 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.506000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.506227 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:44.506093 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-serving-cert: secret "maas-api-e2e-ait-5f53f6be-serving-cert" not found Sep 16 14:27:44.506227 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:44.506174 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls podName:003c44f8-1e95-491b-b8df-027a1be19642 nodeName:}" failed. No retries permitted until 2026-09-16 14:27:45.006151861 +0000 UTC m=+1195.583512994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls") pod "maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" (UID: "003c44f8-1e95-491b-b8df-027a1be19642") : secret "maas-api-e2e-ait-5f53f6be-serving-cert" not found Sep 16 14:27:44.506673 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.506651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.508700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.508682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.530087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.530046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgf6\" (UniqueName: \"kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:44.639269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.639179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:44.778590 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:44.778559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:45.012121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.012086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:45.015029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.015004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") pod \"maas-api-e2e-ait-5f53f6be-748cd79d4-592p4\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:45.226910 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.226863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:45.229046 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.229014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:45.335403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.335356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:45.518869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.518835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:27:45.596972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.596941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" event={"ID":"003c44f8-1e95-491b-b8df-027a1be19642","Type":"ContainerStarted","Data":"9a81f93710256f692782561dbc9950885669a287063ef49b167b9849ffd9e4a4"} Sep 16 14:27:45.794532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.794499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:45.835307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.835240 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:45.924023 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.923995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924041 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr88x\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924210 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924252 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924286 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert\") pod \"09c169a9-29a5-423f-a391-f2d328001baa\" (UID: \"09c169a9-29a5-423f-a391-f2d328001baa\") " Sep 16 14:27:45.924643 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data" (OuterVolumeSpecName: "istio-data") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:45.924716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924697 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:45.925028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924843 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-credential-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:45.925028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924866 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-data\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:45.925028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.924887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:45.925028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.925001 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:45.925445 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.925106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:27:45.927400 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.927365 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:27:45.927503 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.927466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:27:45.931202 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.929086 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x" (OuterVolumeSpecName: "kube-api-access-gr88x") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "kube-api-access-gr88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:45.931202 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:45.929354 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token" (OuterVolumeSpecName: "istio-token") pod "09c169a9-29a5-423f-a391-f2d328001baa" (UID: "09c169a9-29a5-423f-a391-f2d328001baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:27:46.026209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-istio-envoy\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026208 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr88x\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-kube-api-access-gr88x\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026219 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c169a9-29a5-423f-a391-f2d328001baa-istio-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026227 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026236 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c169a9-29a5-423f-a391-f2d328001baa-istio-podinfo\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026245 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c169a9-29a5-423f-a391-f2d328001baa-workload-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.026416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.026254 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c169a9-29a5-423f-a391-f2d328001baa-istiod-ca-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:27:46.605139 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.605105 2579 generic.go:356] "Generic (PLEG): container finished" podID="09c169a9-29a5-423f-a391-f2d328001baa" containerID="d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a" exitCode=0 Sep 16 14:27:46.605584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.605175 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" Sep 16 14:27:46.605584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.605190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerDied","Data":"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a"} Sep 16 14:27:46.605584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.605290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s" event={"ID":"09c169a9-29a5-423f-a391-f2d328001baa","Type":"ContainerDied","Data":"03f267bfdc848a11a4a09e24f9e22f9c0e7ef66ee0f6f64f8fe14a2282abe4a9"} Sep 16 14:27:46.605584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.605313 2579 scope.go:117] "RemoveContainer" containerID="d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a" Sep 16 14:27:46.606953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.606929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" event={"ID":"003c44f8-1e95-491b-b8df-027a1be19642","Type":"ContainerStarted","Data":"0693cbcaf9fd5fb85c5cca646d847237156b9b164b17275251c97bfb42c050f4"} Sep 16 14:27:46.607035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.606984 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:27:46.619999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.619980 2579 scope.go:117] "RemoveContainer" containerID="8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444" Sep 16 14:27:46.630951 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.630887 2579 scope.go:117] "RemoveContainer" containerID="d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a" Sep 16 14:27:46.631217 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:46.631195 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a\": container with ID starting with d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a not found: ID does not exist" containerID="d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a" Sep 16 14:27:46.631301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.631225 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a"} err="failed to get container status \"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a\": rpc error: code = NotFound desc = could not find container \"d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a\": container with ID starting with d819105f549a272cb4d28aca9e7c606732d31a56bd05ada435f520031b5e813a not found: ID does not exist" Sep 16 14:27:46.631301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.631243 2579 scope.go:117] "RemoveContainer" containerID="8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444" Sep 16 14:27:46.631558 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:27:46.631533 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444\": container with ID starting with 8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444 not found: ID does not exist" containerID="8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444" Sep 16 14:27:46.631628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.631563 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444"} err="failed to get container status \"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444\": rpc error: code = NotFound desc = could not find container \"8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444\": container with ID starting with 8e768eb0e6973bd1518b3655676bf3442576aef65d46897a43a67477b0aba444 not found: ID does not exist" Sep 16 14:27:46.640637 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.640618 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:46.644240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.644222 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-beaf471d-gw-openshift-default-764c49c568-kpm5s"] Sep 16 14:27:46.691398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:46.691345 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" podStartSLOduration=2.361113486 podStartE2EDuration="2.691324479s" totalImagesPullingTime="330.210993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:46.687241388 +0000 UTC m=+1197.264602540" watchObservedRunningTime="2026-09-16 14:27:46.691324479 +0000 UTC m=+1197.268685632" Sep 16 14:27:47.583053 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:47.583022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:27:47.817987 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:47.817957 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c169a9-29a5-423f-a391-f2d328001baa" path="/var/lib/kubelet/pods/09c169a9-29a5-423f-a391-f2d328001baa/volumes" Sep 16 14:27:48.441140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:48.441105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:49.375743 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:49.375709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:27:49.894310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:49.894283 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:27:49.897928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:49.897907 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:27:52.619752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:27:52.619722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:28:01.033430 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:01.033388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:03.175262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:03.175225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:04.162406 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:04.162372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:04.569062 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:04.569020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:04.669487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:04.669438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:04.754578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:04.754546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:05.006092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:05.006053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:05.150477 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:05.150434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:05.343705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:05.343527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:05.514487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:05.514455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:06.251000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:06.250955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:07.234624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:07.234562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:08.488972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:08.488928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:08.807479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:08.807340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:08.906722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:08.906675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:09.759693 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:09.759652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:10.869396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:10.869363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:11.719116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719085 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:28:11.719794 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719770 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:11.719794 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719791 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:11.720035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719804 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:11.720035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719810 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:11.720035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.719886 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:11.723218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.723196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.726346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.726324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d31d7c-serving-cert\"" Sep 16 14:28:11.736971 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.736949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:28:11.757026 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.756983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:11.776219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.776196 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerID="6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12" exitCode=0 Sep 16 14:28:11.776327 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.776250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerDied","Data":"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12"} Sep 16 14:28:11.776687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.776660 2579 scope.go:117] "RemoveContainer" containerID="6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12" Sep 16 14:28:11.777844 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.777814 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:28:11.777925 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.777879 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:28:11.777925 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.777913 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:28:11.790434 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.790406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8q9\" (UniqueName: \"kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.790596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.790576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.790695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.790641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.790695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.790679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.810950 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.810928 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:28:11.891332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.891302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.891332 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.891336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.891836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.891358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.891836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.891428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8q9\" (UniqueName: \"kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.891836 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:28:11.891480 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-serving-cert: secret "maas-api-e2e-shared-b-w2-d31d7c-serving-cert" not found Sep 16 14:28:11.891836 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:28:11.891552 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls podName:0a3bfefa-6d77-43c3-8210-494a8a5271e0 nodeName:}" failed. No retries permitted until 2026-09-16 14:28:12.391531696 +0000 UTC m=+1222.968892829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" (UID: "0a3bfefa-6d77-43c3-8210-494a8a5271e0") : secret "maas-api-e2e-shared-b-w2-d31d7c-serving-cert" not found Sep 16 14:28:11.892239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.892217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.894204 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.894176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:11.903270 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:11.903247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8q9\" (UniqueName: \"kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:12.087467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.087387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:12.187927 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.187894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:12.295012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.294981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:12.380214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.380107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:12.399116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.397969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:12.404415 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.404368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:12.638634 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.638503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:12.786278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.786239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerStarted","Data":"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af"} Sep 16 14:28:12.920132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.920036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:12.957296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:12.940548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:28:13.552752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.552664 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:28:13.553034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.553004 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:28:13.553158 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.553075 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:28:13.794747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.794654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" event={"ID":"0a3bfefa-6d77-43c3-8210-494a8a5271e0","Type":"ContainerStarted","Data":"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d"} Sep 16 14:28:13.794903 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.794857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:13.794959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.794900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" event={"ID":"0a3bfefa-6d77-43c3-8210-494a8a5271e0","Type":"ContainerStarted","Data":"80968823346d47a75bc7a22d54fa0cfc507b91e8f1632dc86a5a707098ebbf71"} Sep 16 14:28:13.814342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:13.814242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" podStartSLOduration=2.329021489 podStartE2EDuration="2.814227164s" totalImagesPullingTime="485.205675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:13.81241481 +0000 UTC m=+1224.389775964" watchObservedRunningTime="2026-09-16 14:28:13.814227164 +0000 UTC m=+1224.391588316" Sep 16 14:28:14.132648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:14.132526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:14.553116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:14.553070 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:28:14.553322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:14.553140 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:28:14.944216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:14.944173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:15.553210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.553170 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:28:15.553899 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.553852 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:28:15.669432 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.669399 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:28:15.669700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.669674 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" podUID="003c44f8-1e95-491b-b8df-027a1be19642" containerName="maas-api" containerID="cri-o://0693cbcaf9fd5fb85c5cca646d847237156b9b164b17275251c97bfb42c050f4" gracePeriod=30 Sep 16 14:28:15.706669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.706626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:15.796479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.796405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:15.811728 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.811641 2579 generic.go:356] "Generic (PLEG): container finished" podID="003c44f8-1e95-491b-b8df-027a1be19642" containerID="0693cbcaf9fd5fb85c5cca646d847237156b9b164b17275251c97bfb42c050f4" exitCode=0 Sep 16 14:28:15.820279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.820240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" event={"ID":"003c44f8-1e95-491b-b8df-027a1be19642","Type":"ContainerDied","Data":"0693cbcaf9fd5fb85c5cca646d847237156b9b164b17275251c97bfb42c050f4"} Sep 16 14:28:15.899799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.899761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:15.993484 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:15.993463 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:28:16.004399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.004371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:16.062907 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.062813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbgf6\" (UniqueName: \"kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6\") pod \"003c44f8-1e95-491b-b8df-027a1be19642\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " Sep 16 14:28:16.063080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.062917 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") pod \"003c44f8-1e95-491b-b8df-027a1be19642\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " Sep 16 14:28:16.063080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.062998 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca\") pod \"003c44f8-1e95-491b-b8df-027a1be19642\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " Sep 16 14:28:16.063080 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.063030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs\") pod \"003c44f8-1e95-491b-b8df-027a1be19642\" (UID: \"003c44f8-1e95-491b-b8df-027a1be19642\") " Sep 16 14:28:16.063536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.063514 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "003c44f8-1e95-491b-b8df-027a1be19642" (UID: "003c44f8-1e95-491b-b8df-027a1be19642"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:28:16.065678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.065641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "003c44f8-1e95-491b-b8df-027a1be19642" (UID: "003c44f8-1e95-491b-b8df-027a1be19642"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:28:16.065788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.065716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "003c44f8-1e95-491b-b8df-027a1be19642" (UID: "003c44f8-1e95-491b-b8df-027a1be19642"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:28:16.065856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.065822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6" (OuterVolumeSpecName: "kube-api-access-zbgf6") pod "003c44f8-1e95-491b-b8df-027a1be19642" (UID: "003c44f8-1e95-491b-b8df-027a1be19642"). InnerVolumeSpecName "kube-api-access-zbgf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:28:16.145631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.144197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:16.164321 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.164289 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/003c44f8-1e95-491b-b8df-027a1be19642-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:16.164321 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.164323 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:16.164573 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.164342 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgf6\" (UniqueName: \"kubernetes.io/projected/003c44f8-1e95-491b-b8df-027a1be19642-kube-api-access-zbgf6\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:16.164573 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.164356 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/003c44f8-1e95-491b-b8df-027a1be19642-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:16.552467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.552430 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:28:16.552823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.552779 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:28:16.655476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.655437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:16.819662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.819567 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" Sep 16 14:28:16.819662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.819567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4" event={"ID":"003c44f8-1e95-491b-b8df-027a1be19642","Type":"ContainerDied","Data":"9a81f93710256f692782561dbc9950885669a287063ef49b167b9849ffd9e4a4"} Sep 16 14:28:16.819882 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.819685 2579 scope.go:117] "RemoveContainer" containerID="0693cbcaf9fd5fb85c5cca646d847237156b9b164b17275251c97bfb42c050f4" Sep 16 14:28:16.850563 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.850521 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:28:16.861683 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:16.861642 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f53f6be-748cd79d4-592p4"] Sep 16 14:28:17.223437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.223402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:17.552579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.552478 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 14:28:17.552579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.552563 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 14:28:17.701565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.701534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:17.818038 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.817957 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003c44f8-1e95-491b-b8df-027a1be19642" path="/var/lib/kubelet/pods/003c44f8-1e95-491b-b8df-027a1be19642/volumes" Sep 16 14:28:17.967465 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:17.967423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:18.083981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:18.083862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:18.247659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:18.247623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:18.556217 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:18.556189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:28:18.556650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:18.556633 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:28:18.557337 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:18.557318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:28:19.060575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:19.060543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:19.807326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:19.807288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:28:20.551487 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:20.551456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:21.166449 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:21.166406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:22.445013 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.444978 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:28:22.445453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.445440 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003c44f8-1e95-491b-b8df-027a1be19642" containerName="maas-api" Sep 16 14:28:22.445453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.445454 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="003c44f8-1e95-491b-b8df-027a1be19642" containerName="maas-api" Sep 16 14:28:22.445546 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.445538 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c169a9-29a5-423f-a391-f2d328001baa" containerName="istio-proxy" Sep 16 14:28:22.445588 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.445551 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="003c44f8-1e95-491b-b8df-027a1be19642" containerName="maas-api" Sep 16 14:28:22.448592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.448577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.453057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.453036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-377cc0d5-serving-cert\"" Sep 16 14:28:22.490840 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.490815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:28:22.526118 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.526087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:22.532366 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.532336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.532510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.532415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.532510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.532457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.532619 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.532575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7x76\" (UniqueName: \"kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.633994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.633964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7x76\" (UniqueName: \"kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.634167 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.634018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.634167 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.634073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.634167 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.634106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.634366 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:28:22.634244 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-serving-cert: secret "maas-api-e2e-derive-377cc0d5-serving-cert" not found Sep 16 14:28:22.634366 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:28:22.634307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls podName:16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9 nodeName:}" failed. No retries permitted until 2026-09-16 14:28:23.134288798 +0000 UTC m=+1233.711649941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls") pod "maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" (UID: "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9") : secret "maas-api-e2e-derive-377cc0d5-serving-cert" not found Sep 16 14:28:22.635036 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.634956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.636816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.636791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.654650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.654599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7x76\" (UniqueName: \"kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:22.695675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.695587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:22.988811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:22.988702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:23.133852 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.133814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:23.139864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.139832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:23.142856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.142829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") pod \"maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:23.234656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.234583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:23.359404 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.359304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:23.633952 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.633861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:23.647018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.646990 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:28:23.648524 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:28:23.648497 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e917a1_c9b9_4b6a_b549_6cd13cbbc8b9.slice/crio-e8266b06ee54bc7365641d4fcdda24afb3158aa203be948a820061f0cae5beb5 WatchSource:0}: Error finding container e8266b06ee54bc7365641d4fcdda24afb3158aa203be948a820061f0cae5beb5: Status 404 returned error can't find the container with id e8266b06ee54bc7365641d4fcdda24afb3158aa203be948a820061f0cae5beb5 Sep 16 14:28:23.868758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:23.868720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" event={"ID":"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9","Type":"ContainerStarted","Data":"e8266b06ee54bc7365641d4fcdda24afb3158aa203be948a820061f0cae5beb5"} Sep 16 14:28:24.827626 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:24.827502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:24.877128 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:24.877082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" event={"ID":"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9","Type":"ContainerStarted","Data":"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae"} Sep 16 14:28:24.877302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:24.877229 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:24.896887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:24.896835 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" podStartSLOduration=2.070396473 podStartE2EDuration="2.896817307s" totalImagesPullingTime="826.420834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:24.89486252 +0000 UTC m=+1235.472223675" watchObservedRunningTime="2026-09-16 14:28:24.896817307 +0000 UTC m=+1235.474178458" Sep 16 14:28:24.908659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:24.908597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:25.853156 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:25.853123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:26.755947 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:26.755901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:27.630497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:27.630461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:28.334716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:28.334681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:28.657131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:28.657043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:29.422083 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:29.422038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:29.973954 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:29.973904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:30.889287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:30.889260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:28:32.822807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:32.822767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:33.822999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:33.822961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:35.156529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:35.156492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:40.920121 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:40.920082 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:28:40.920569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:40.920349 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" podUID="464ea653-88af-409c-9b5b-57f5f008f689" containerName="maas-api" containerID="cri-o://798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8" gracePeriod=30 Sep 16 14:28:40.953943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:40.953913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:41.183166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.183143 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:28:41.205980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.205943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:41.223666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.223644 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs\") pod \"464ea653-88af-409c-9b5b-57f5f008f689\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " Sep 16 14:28:41.223801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.223677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28m8m\" (UniqueName: \"kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m\") pod \"464ea653-88af-409c-9b5b-57f5f008f689\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " Sep 16 14:28:41.223801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.223730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca\") pod \"464ea653-88af-409c-9b5b-57f5f008f689\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " Sep 16 14:28:41.223801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.223750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls\") pod \"464ea653-88af-409c-9b5b-57f5f008f689\" (UID: \"464ea653-88af-409c-9b5b-57f5f008f689\") " Sep 16 14:28:41.224134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.224105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "464ea653-88af-409c-9b5b-57f5f008f689" (UID: "464ea653-88af-409c-9b5b-57f5f008f689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:28:41.225964 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.225934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "464ea653-88af-409c-9b5b-57f5f008f689" (UID: "464ea653-88af-409c-9b5b-57f5f008f689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:28:41.226069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.226014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "464ea653-88af-409c-9b5b-57f5f008f689" (UID: "464ea653-88af-409c-9b5b-57f5f008f689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:28:41.226125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.226103 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m" (OuterVolumeSpecName: "kube-api-access-28m8m") pod "464ea653-88af-409c-9b5b-57f5f008f689" (UID: "464ea653-88af-409c-9b5b-57f5f008f689"). InnerVolumeSpecName "kube-api-access-28m8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:28:41.325450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.325413 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:41.325450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.325443 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28m8m\" (UniqueName: \"kubernetes.io/projected/464ea653-88af-409c-9b5b-57f5f008f689-kube-api-access-28m8m\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:41.325450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.325454 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464ea653-88af-409c-9b5b-57f5f008f689-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:41.325734 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.325463 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464ea653-88af-409c-9b5b-57f5f008f689-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:28:41.407172 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.407085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:41.595233 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.595140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:41.971476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.971439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:41.988064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.988039 2579 generic.go:356] "Generic (PLEG): container finished" podID="464ea653-88af-409c-9b5b-57f5f008f689" containerID="798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8" exitCode=0 Sep 16 14:28:41.988227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.988091 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" Sep 16 14:28:41.988227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.988130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" event={"ID":"464ea653-88af-409c-9b5b-57f5f008f689","Type":"ContainerDied","Data":"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8"} Sep 16 14:28:41.988227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.988184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v" event={"ID":"464ea653-88af-409c-9b5b-57f5f008f689","Type":"ContainerDied","Data":"06063acbf1d71fdfe1c476dfcc7d59147a31ae69cd61899f61fdade219d43473"} Sep 16 14:28:41.988227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.988216 2579 scope.go:117] "RemoveContainer" containerID="798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8" Sep 16 14:28:41.999133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.999111 2579 scope.go:117] "RemoveContainer" containerID="798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8" Sep 16 14:28:41.999396 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:28:41.999377 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8\": container with ID starting with 798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8 not found: ID does not exist" containerID="798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8" Sep 16 14:28:41.999459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:41.999405 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8"} err="failed to get container status \"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8\": rpc error: code = NotFound desc = could not find container \"798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8\": container with ID starting with 798d49f41abc40a32e2acb364946cd73adbb0de8a622dc63400d78f39e5362b8 not found: ID does not exist" Sep 16 14:28:42.012439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:42.012418 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:28:42.018101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:42.018082 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d48266-6cb4d85749-s6g7v"] Sep 16 14:28:42.765970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:42.765936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:43.719023 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:43.718991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:43.817772 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:43.817730 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464ea653-88af-409c-9b5b-57f5f008f689" path="/var/lib/kubelet/pods/464ea653-88af-409c-9b5b-57f5f008f689/volumes" Sep 16 14:28:44.573164 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:44.573128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:45.505829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:45.505787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:46.526499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:46.526454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:47.216583 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:47.216551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:48.180405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:48.180362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:48.815635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:48.815592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:49.131731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:49.131637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:49.235752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:49.235719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:28:49.380545 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:28:49.380510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:02.185374 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.185342 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:29:02.185913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.185630 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" podUID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" containerName="maas-api" containerID="cri-o://944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae" gracePeriod=30 Sep 16 14:29:02.228152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.228069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:02.386547 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.386163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:02.459303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.459281 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:29:02.530452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.530063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:02.551132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.551104 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs\") pod \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " Sep 16 14:29:02.551294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.551159 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7x76\" (UniqueName: \"kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76\") pod \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " Sep 16 14:29:02.551294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.551181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") pod \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " Sep 16 14:29:02.551294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.551231 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca\") pod \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\" (UID: \"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9\") " Sep 16 14:29:02.551658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.551633 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" (UID: "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:29:02.553431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.553408 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" (UID: "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:29:02.553926 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.553906 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" (UID: "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:29:02.553926 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.553913 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76" (OuterVolumeSpecName: "kube-api-access-g7x76") pod "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" (UID: "16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9"). InnerVolumeSpecName "kube-api-access-g7x76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:29:02.606775 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.606736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:02.652498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.652454 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7x76\" (UniqueName: \"kubernetes.io/projected/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-kube-api-access-g7x76\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:29:02.652498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.652493 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:29:02.652801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.652510 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:29:02.652801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.652523 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:29:02.716088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:02.715999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:03.129804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.129713 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" containerID="944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae" exitCode=0 Sep 16 14:29:03.129804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.129757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" event={"ID":"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9","Type":"ContainerDied","Data":"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae"} Sep 16 14:29:03.129804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.129764 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" Sep 16 14:29:03.130092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.129811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk" event={"ID":"16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9","Type":"ContainerDied","Data":"e8266b06ee54bc7365641d4fcdda24afb3158aa203be948a820061f0cae5beb5"} Sep 16 14:29:03.130092 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.129842 2579 scope.go:117] "RemoveContainer" containerID="944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae" Sep 16 14:29:03.141425 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.141409 2579 scope.go:117] "RemoveContainer" containerID="944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae" Sep 16 14:29:03.141714 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:29:03.141692 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae\": container with ID starting with 944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae not found: ID does not exist" containerID="944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae" Sep 16 14:29:03.141815 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.141718 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae"} err="failed to get container status \"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae\": rpc error: code = NotFound desc = could not find container \"944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae\": container with ID starting with 944d4646893e523a0e4525c5a11688c9f0b9da6a1621dd580fb41db2694723ae not found: ID does not exist" Sep 16 14:29:03.163970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.163942 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:29:03.170944 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.170922 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-377cc0d5-7c687d9c8f-jpwkk"] Sep 16 14:29:03.293688 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.293654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:03.427527 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.427485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:03.581978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.581947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:03.817526 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:03.817451 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" path="/var/lib/kubelet/pods/16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9/volumes" Sep 16 14:29:04.236097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:04.236052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:04.999057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:04.999030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:05.186078 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:05.186041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:05.542731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:05.542683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:05.642310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:05.642278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:06.837908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:06.837873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:11.193809 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:11.193770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:11.316117 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:11.316084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:11.414362 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:11.414329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:11.517254 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:11.517155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:12.059222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:12.059187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:13.811851 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:13.811820 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:29:36.512399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:36.512363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:36.821450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:36.821358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:37.905109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:37.905073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:39.089696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:39.089662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:39.176322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:39.176280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:40.624702 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:40.624665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:43.532600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:43.532549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:44.103812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:44.103761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:44.216739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:44.216693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:29:44.304632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:29:44.304572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:00.256113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:00.256080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:00.817118 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:00.817066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:06.706408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:06.706371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:06.812089 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:06.812050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:07.159331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:07.159242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:17.480407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:17.480361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:17.811400 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:17.811308 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:30:18.289833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:18.289794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:24.782042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:24.782000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:24.911554 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:24.911521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:25.007264 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:25.007224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:25.111890 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:25.111795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:25.826845 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:25.826807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:25.923561 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:25.923529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:26.023791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:26.023759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:27.557467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:27.557431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:33.662392 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:33.662354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:33.762017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:33.761979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:34.386025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:34.385986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:40.458584 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:40.458538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:40.576370 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:40.576341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:40.722401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:40.722315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:41.129088 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:41.128997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:41.429476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:41.429436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:43.219999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:43.219963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:43.965283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:43.965249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:44.059513 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:44.059474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:44.196192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:44.196161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:45.148671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.148630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:45.925484 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.925451 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:30:45.925908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.925895 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" containerName="maas-api" Sep 16 14:30:45.925957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.925910 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" containerName="maas-api" Sep 16 14:30:45.925957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.925920 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464ea653-88af-409c-9b5b-57f5f008f689" containerName="maas-api" Sep 16 14:30:45.925957 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.925926 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="464ea653-88af-409c-9b5b-57f5f008f689" containerName="maas-api" Sep 16 14:30:45.926053 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.926005 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="464ea653-88af-409c-9b5b-57f5f008f689" containerName="maas-api" Sep 16 14:30:45.926053 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.926015 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e917a1-c9b9-4b6a-b549-6cd13cbbc8b9" containerName="maas-api" Sep 16 14:30:45.929683 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.929659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:45.932386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.932367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2524a893-serving-cert\"" Sep 16 14:30:45.942328 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.942301 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:30:45.949240 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:45.948393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:46.045191 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.045153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:46.048348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.048326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.048452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.048360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.048452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.048416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.048452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.048442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mmkh\" (UniqueName: \"kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.143481 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.143439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:46.148947 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.148924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.149383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.148957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.149383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.148998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.149383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.149024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mmkh\" (UniqueName: \"kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.149808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.149735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.151628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.151580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.151628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.151599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.165579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.165554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mmkh\" (UniqueName: \"kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh\") pod \"maas-api-e2e-mt-2524a893-5c794dc885-l5x8j\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.241497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.241425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:46.532704 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.532674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:30:46.533712 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:30:46.533685 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48e751b_d237_43c1_9c8e_51dbbf315943.slice/crio-a65e94c9f902faad336489bb0278a463c85198477e86fd3ba4ccf588fa6ff61c WatchSource:0}: Error finding container a65e94c9f902faad336489bb0278a463c85198477e86fd3ba4ccf588fa6ff61c: Status 404 returned error can't find the container with id a65e94c9f902faad336489bb0278a463c85198477e86fd3ba4ccf588fa6ff61c Sep 16 14:30:46.535498 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.535480 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:30:46.779111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:46.779078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" event={"ID":"d48e751b-d237-43c1-9c8e-51dbbf315943","Type":"ContainerStarted","Data":"a65e94c9f902faad336489bb0278a463c85198477e86fd3ba4ccf588fa6ff61c"} Sep 16 14:30:47.119275 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:47.119242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:47.283115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:47.283086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:47.786995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:47.786959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" event={"ID":"d48e751b-d237-43c1-9c8e-51dbbf315943","Type":"ContainerStarted","Data":"c44025b40a8abec7969bce36488fa7958d74353c39762782223eb0bafdeab029"} Sep 16 14:30:47.787201 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:47.787045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:30:47.811995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:47.811944 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" podStartSLOduration=2.084845097 podStartE2EDuration="2.811927943s" totalImagesPullingTime="727.082846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:47.810060053 +0000 UTC m=+1378.387421202" watchObservedRunningTime="2026-09-16 14:30:47.811927943 +0000 UTC m=+1378.389289094" Sep 16 14:30:48.392871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:48.392838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:48.514465 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:48.514429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:30:53.806075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:30:53.806045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:31:03.792399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:03.792361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:04.084314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:04.084231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:05.307962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:05.307919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:06.200407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:06.200371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:07.149999 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:07.149963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:07.906222 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:07.906183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:08.809427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:08.809396 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:31:08.904825 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:08.904790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:09.005454 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:09.005377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:09.153203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:09.153098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:09.533979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:09.533941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:09.637631 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:09.637573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:09.991659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:09.991630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:10.315448 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:10.315351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:11.225410 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:11.225377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:12.052176 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:12.052145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:16.279998 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:16.279960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:16.385244 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:16.385211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:17.052665 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:17.052628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.197304 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.197271 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:31:18.197797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.197522 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" podUID="ef293318-eeb1-4b97-871d-91db6db018ef" containerName="maas-api" containerID="cri-o://8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c" gracePeriod=30 Sep 16 14:31:18.252569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.252530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.362116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.362083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.466132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.466094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.466713 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.466695 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:31:18.577443 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.577414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca\") pod \"ef293318-eeb1-4b97-871d-91db6db018ef\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " Sep 16 14:31:18.577632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.577483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlpws\" (UniqueName: \"kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws\") pod \"ef293318-eeb1-4b97-871d-91db6db018ef\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " Sep 16 14:31:18.577632 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.577519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") pod \"ef293318-eeb1-4b97-871d-91db6db018ef\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " Sep 16 14:31:18.577715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.577677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs\") pod \"ef293318-eeb1-4b97-871d-91db6db018ef\" (UID: \"ef293318-eeb1-4b97-871d-91db6db018ef\") " Sep 16 14:31:18.577844 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.577824 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef293318-eeb1-4b97-871d-91db6db018ef" (UID: "ef293318-eeb1-4b97-871d-91db6db018ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:31:18.578048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.578029 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef293318-eeb1-4b97-871d-91db6db018ef-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:18.579888 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.579859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws" (OuterVolumeSpecName: "kube-api-access-tlpws") pod "ef293318-eeb1-4b97-871d-91db6db018ef" (UID: "ef293318-eeb1-4b97-871d-91db6db018ef"). InnerVolumeSpecName "kube-api-access-tlpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:31:18.579888 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.579878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef293318-eeb1-4b97-871d-91db6db018ef" (UID: "ef293318-eeb1-4b97-871d-91db6db018ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:31:18.580016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.579864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef293318-eeb1-4b97-871d-91db6db018ef" (UID: "ef293318-eeb1-4b97-871d-91db6db018ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:31:18.633828 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.633798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.679533 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.679500 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:18.679533 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.679530 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlpws\" (UniqueName: \"kubernetes.io/projected/ef293318-eeb1-4b97-871d-91db6db018ef-kube-api-access-tlpws\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:18.679755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.679545 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef293318-eeb1-4b97-871d-91db6db018ef-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:18.776829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.776726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:18.996301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.996269 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef293318-eeb1-4b97-871d-91db6db018ef" containerID="8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c" exitCode=0 Sep 16 14:31:18.996473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.996324 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" Sep 16 14:31:18.996473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.996358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" event={"ID":"ef293318-eeb1-4b97-871d-91db6db018ef","Type":"ContainerDied","Data":"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c"} Sep 16 14:31:18.996473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.996460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j" event={"ID":"ef293318-eeb1-4b97-871d-91db6db018ef","Type":"ContainerDied","Data":"08b49651d4f3fe25c85b7dd29d55cb5641cba3bd065044ce3ff1b14fab83e38e"} Sep 16 14:31:18.996596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:18.996493 2579 scope.go:117] "RemoveContainer" containerID="8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c" Sep 16 14:31:19.007790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.007515 2579 scope.go:117] "RemoveContainer" containerID="8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c" Sep 16 14:31:19.008001 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:31:19.007972 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c\": container with ID starting with 8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c not found: ID does not exist" containerID="8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c" Sep 16 14:31:19.008060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.008014 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c"} err="failed to get container status \"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c\": rpc error: code = NotFound desc = could not find container \"8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c\": container with ID starting with 8f661f6ee1b538275663ed5b589f0f3393937e8a53cf4b015ea5c3c0f0a0323c not found: ID does not exist" Sep 16 14:31:19.028316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.028253 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:31:19.033184 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.033143 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-11636c-7fd89f8dbd-6mg2j"] Sep 16 14:31:19.052789 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.052759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:19.497952 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.497921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:19.818556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:19.818473 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef293318-eeb1-4b97-871d-91db6db018ef" path="/var/lib/kubelet/pods/ef293318-eeb1-4b97-871d-91db6db018ef/volumes" Sep 16 14:31:20.305877 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:20.305848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:21.449310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:21.449281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:23.830534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:23.830494 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" podUID="527f6558-dcde-4c68-9c32-1abcddf90f9b" containerName="main" containerID="cri-o://737698b2aa17bd3d5425b0b4aed7db163a734fd954f4a79d8740f7f48ec613b6" gracePeriod=45 Sep 16 14:31:24.044009 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.043975 2579 generic.go:356] "Generic (PLEG): container finished" podID="527f6558-dcde-4c68-9c32-1abcddf90f9b" containerID="737698b2aa17bd3d5425b0b4aed7db163a734fd954f4a79d8740f7f48ec613b6" exitCode=0 Sep 16 14:31:24.044179 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.044044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" event={"ID":"527f6558-dcde-4c68-9c32-1abcddf90f9b","Type":"ContainerDied","Data":"737698b2aa17bd3d5425b0b4aed7db163a734fd954f4a79d8740f7f48ec613b6"} Sep 16 14:31:24.101624 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.101587 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:31:24.239928 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.239893 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240036 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240092 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz\") pod \"527f6558-dcde-4c68-9c32-1abcddf90f9b\" (UID: \"527f6558-dcde-4c68-9c32-1abcddf90f9b\") " Sep 16 14:31:24.240415 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:31:24.240544 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240513 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:31:24.240729 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240577 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-tmp-dir\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:24.240824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.240803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home" (OuterVolumeSpecName: "home") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:31:24.242770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.242744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz" (OuterVolumeSpecName: "kube-api-access-qjzrz") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "kube-api-access-qjzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:31:24.243097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.243079 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:31:24.243160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.243087 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm" (OuterVolumeSpecName: "dshm") pod "527f6558-dcde-4c68-9c32-1abcddf90f9b" (UID: "527f6558-dcde-4c68-9c32-1abcddf90f9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:31:24.341661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.341564 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/527f6558-dcde-4c68-9c32-1abcddf90f9b-kube-api-access-qjzrz\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:24.341661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.341594 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527f6558-dcde-4c68-9c32-1abcddf90f9b-tls-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:24.341661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.341618 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-home\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:24.341661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.341630 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-model-cache\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:24.341661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:24.341642 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527f6558-dcde-4c68-9c32-1abcddf90f9b-dshm\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:25.052056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.052026 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" Sep 16 14:31:25.052056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.052039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv" event={"ID":"527f6558-dcde-4c68-9c32-1abcddf90f9b","Type":"ContainerDied","Data":"8224667ba9b6fc99a1ee981217350c81751cfdbee2c71a9a9d7995d0aed8f678"} Sep 16 14:31:25.052562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.052091 2579 scope.go:117] "RemoveContainer" containerID="737698b2aa17bd3d5425b0b4aed7db163a734fd954f4a79d8740f7f48ec613b6" Sep 16 14:31:25.083641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.083590 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:31:25.086968 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.086948 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-11636c/premium-w0-11636c-kserve-6c749d8cc8-k8djv"] Sep 16 14:31:25.817625 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.817579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:25.818875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:25.818852 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="527f6558-dcde-4c68-9c32-1abcddf90f9b" path="/var/lib/kubelet/pods/527f6558-dcde-4c68-9c32-1abcddf90f9b/volumes" Sep 16 14:31:26.069948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:26.069858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:26.325676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:26.325574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:26.982589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:26.982551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:27.886800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:27.886771 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:31:27.887201 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:27.886963 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" podUID="d48e751b-d237-43c1-9c8e-51dbbf315943" containerName="maas-api" containerID="cri-o://c44025b40a8abec7969bce36488fa7958d74353c39762782223eb0bafdeab029" gracePeriod=30 Sep 16 14:31:27.948989 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:27.948939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.077289 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.077255 2579 generic.go:356] "Generic (PLEG): container finished" podID="d48e751b-d237-43c1-9c8e-51dbbf315943" containerID="c44025b40a8abec7969bce36488fa7958d74353c39762782223eb0bafdeab029" exitCode=0 Sep 16 14:31:28.077464 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.077377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" event={"ID":"d48e751b-d237-43c1-9c8e-51dbbf315943","Type":"ContainerDied","Data":"c44025b40a8abec7969bce36488fa7958d74353c39762782223eb0bafdeab029"} Sep 16 14:31:28.083813 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.083771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.141241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.141183 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:31:28.219769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.219740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.277563 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.277536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca\") pod \"d48e751b-d237-43c1-9c8e-51dbbf315943\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " Sep 16 14:31:28.277751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.277581 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mmkh\" (UniqueName: \"kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh\") pod \"d48e751b-d237-43c1-9c8e-51dbbf315943\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " Sep 16 14:31:28.277751 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.277725 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls\") pod \"d48e751b-d237-43c1-9c8e-51dbbf315943\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " Sep 16 14:31:28.277878 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.277767 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs\") pod \"d48e751b-d237-43c1-9c8e-51dbbf315943\" (UID: \"d48e751b-d237-43c1-9c8e-51dbbf315943\") " Sep 16 14:31:28.277989 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.277968 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d48e751b-d237-43c1-9c8e-51dbbf315943" (UID: "d48e751b-d237-43c1-9c8e-51dbbf315943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:31:28.278114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.278094 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e751b-d237-43c1-9c8e-51dbbf315943-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:28.279973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.279948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d48e751b-d237-43c1-9c8e-51dbbf315943" (UID: "d48e751b-d237-43c1-9c8e-51dbbf315943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:31:28.280094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.279980 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh" (OuterVolumeSpecName: "kube-api-access-6mmkh") pod "d48e751b-d237-43c1-9c8e-51dbbf315943" (UID: "d48e751b-d237-43c1-9c8e-51dbbf315943"). InnerVolumeSpecName "kube-api-access-6mmkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:31:28.280094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.280004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d48e751b-d237-43c1-9c8e-51dbbf315943" (UID: "d48e751b-d237-43c1-9c8e-51dbbf315943"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:31:28.347248 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.347210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.379307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.379282 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mmkh\" (UniqueName: \"kubernetes.io/projected/d48e751b-d237-43c1-9c8e-51dbbf315943-kube-api-access-6mmkh\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:28.379307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.379307 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:28.379486 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.379318 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48e751b-d237-43c1-9c8e-51dbbf315943-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:31:28.463027 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.462989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.784958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.784878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:28.811408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.811380 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:31:28.919356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:28.919322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:29.085981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.085888 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" Sep 16 14:31:29.085981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.085910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j" event={"ID":"d48e751b-d237-43c1-9c8e-51dbbf315943","Type":"ContainerDied","Data":"a65e94c9f902faad336489bb0278a463c85198477e86fd3ba4ccf588fa6ff61c"} Sep 16 14:31:29.085981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.085978 2579 scope.go:117] "RemoveContainer" containerID="c44025b40a8abec7969bce36488fa7958d74353c39762782223eb0bafdeab029" Sep 16 14:31:29.115439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.115415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:31:29.127260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.124566 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2524a893-5c794dc885-l5x8j"] Sep 16 14:31:29.818103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:29.818070 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48e751b-d237-43c1-9c8e-51dbbf315943" path="/var/lib/kubelet/pods/d48e751b-d237-43c1-9c8e-51dbbf315943/volumes" Sep 16 14:31:30.481216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:30.481178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:36.512827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:36.512793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:36.647774 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:36.647737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:41.767441 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:41.767349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:42.114185 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:42.114085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:42.204988 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:42.204955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:45.295451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:45.295421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:45.444109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:45.444074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:46.958667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:46.958619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:53.320327 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:53.320278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:58.370653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:58.370587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:58.490782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:58.490746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:58.589331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:58.589300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:31:58.741381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:31:58.741349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:06.355280 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:06.355242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:06.531965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:06.531934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:07.455936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:07.455902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:07.936251 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:07.936215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:08.094466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:08.094432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:11.480673 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:11.480635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:11.676027 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:11.675989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:12.015303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:12.015266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:14.908433 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908402 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:32:14.908901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908887 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef293318-eeb1-4b97-871d-91db6db018ef" containerName="maas-api" Sep 16 14:32:14.908901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908902 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef293318-eeb1-4b97-871d-91db6db018ef" containerName="maas-api" Sep 16 14:32:14.908977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908922 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527f6558-dcde-4c68-9c32-1abcddf90f9b" containerName="main" Sep 16 14:32:14.908977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908933 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="527f6558-dcde-4c68-9c32-1abcddf90f9b" containerName="main" Sep 16 14:32:14.908977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908953 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48e751b-d237-43c1-9c8e-51dbbf315943" containerName="maas-api" Sep 16 14:32:14.908977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.908958 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48e751b-d237-43c1-9c8e-51dbbf315943" containerName="maas-api" Sep 16 14:32:14.909114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.909033 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef293318-eeb1-4b97-871d-91db6db018ef" containerName="maas-api" Sep 16 14:32:14.909114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.909040 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="527f6558-dcde-4c68-9c32-1abcddf90f9b" containerName="main" Sep 16 14:32:14.909114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.909045 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48e751b-d237-43c1-9c8e-51dbbf315943" containerName="maas-api" Sep 16 14:32:14.913791 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.913769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:14.938109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.938068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:14.938483 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:14.938460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-dc3e5b-serving-cert\"" Sep 16 14:32:15.025263 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.025231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhnx\" (UniqueName: \"kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.025432 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.025287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.025432 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.025382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.025513 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.025455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.054762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.054726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:32:15.126517 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.126481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.126757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.126543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhnx\" (UniqueName: \"kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.126757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.126581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.126757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.126656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.127472 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.127446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.129846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.129816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.129973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.129888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.191300 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.191265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhnx\" (UniqueName: \"kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx\") pod \"maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.224803 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.224773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:15.609799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.609711 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:32:15.615597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:15.615545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:15.618690 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:32:15.618665 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a7eade_2b08_4759_9a3d_87f42740ace5.slice/crio-b2984d34871d461a6e81ca1257a4c42269cbc9639cb8c6c0c36904dd3f8aef13 WatchSource:0}: Error finding container b2984d34871d461a6e81ca1257a4c42269cbc9639cb8c6c0c36904dd3f8aef13: Status 404 returned error can't find the container with id b2984d34871d461a6e81ca1257a4c42269cbc9639cb8c6c0c36904dd3f8aef13 Sep 16 14:32:16.375767 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:16.375731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" event={"ID":"92a7eade-2b08-4759-9a3d-87f42740ace5","Type":"ContainerStarted","Data":"b2984d34871d461a6e81ca1257a4c42269cbc9639cb8c6c0c36904dd3f8aef13"} Sep 16 14:32:16.880124 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:16.880078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:17.206299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:17.206262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:17.383450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:17.383414 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" event={"ID":"92a7eade-2b08-4759-9a3d-87f42740ace5","Type":"ContainerStarted","Data":"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea"} Sep 16 14:32:17.383921 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:17.383483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:17.463802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:17.463697 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" podStartSLOduration=2.376343511 podStartE2EDuration="3.463678566s" totalImagesPullingTime="1.087335055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:17.461000604 +0000 UTC m=+1468.038361756" watchObservedRunningTime="2026-09-16 14:32:17.463678566 +0000 UTC m=+1468.041039718" Sep 16 14:32:18.004348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:18.004317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:19.225967 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:19.225939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:20.410628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:20.410579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:23.394799 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:23.394765 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:32:28.636365 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:28.636320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:40.235315 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:40.235281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:40.331876 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:40.331833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:41.263758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:41.263721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:41.483669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:41.483629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:41.909916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:41.909881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:42.509678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:42.509640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:42.820681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:42.820574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:43.812033 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:43.811995 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:32:45.565112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:45.565076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:45.723842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:45.723805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:46.034551 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:46.034503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:46.969716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:46.969664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:48.348955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.348918 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:32:48.353091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.353069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.355691 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.355671 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-bdfd4a-serving-cert\"" Sep 16 14:32:48.381893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.381867 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:32:48.399131 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.399099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:48.445317 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.445284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.445499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.445410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrg7\" (UniqueName: \"kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.445499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.445441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.445499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.445478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.522962 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.522927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:48.546034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.546000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrg7\" (UniqueName: \"kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.546034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.546035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.546230 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.546073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.546230 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.546136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.546314 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:32:48.546250 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-serving-cert: secret "maas-api-e2e-shared-b-w1-bdfd4a-serving-cert" not found Sep 16 14:32:48.546359 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:32:48.546345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls podName:21a58f2a-bd73-40ed-8d14-c03ab7a97a5e nodeName:}" failed. No retries permitted until 2026-09-16 14:32:49.046323228 +0000 UTC m=+1499.623684375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls") pod "maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" (UID: "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e") : secret "maas-api-e2e-shared-b-w1-bdfd4a-serving-cert" not found Sep 16 14:32:48.546865 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.546847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.548778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.548757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.564524 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.564499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrg7\" (UniqueName: \"kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:48.665380 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.665300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:48.968771 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:48.968740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:49.052291 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.052256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:49.054909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.054884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:49.079296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.079263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:49.263853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.263766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:49.573017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.572991 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:32:49.576108 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:32:49.576082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a58f2a_bd73_40ed_8d14_c03ab7a97a5e.slice/crio-7ce4d9e44f588744ddae7b181430886dad49edd1ee32105e03303fee67fda5e3 WatchSource:0}: Error finding container 7ce4d9e44f588744ddae7b181430886dad49edd1ee32105e03303fee67fda5e3: Status 404 returned error can't find the container with id 7ce4d9e44f588744ddae7b181430886dad49edd1ee32105e03303fee67fda5e3 Sep 16 14:32:49.667948 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.667907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:49.947440 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.947413 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:32:49.952113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:49.952082 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:32:50.590043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:50.590009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" event={"ID":"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e","Type":"ContainerStarted","Data":"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37"} Sep 16 14:32:50.590427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:50.590088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" event={"ID":"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e","Type":"ContainerStarted","Data":"7ce4d9e44f588744ddae7b181430886dad49edd1ee32105e03303fee67fda5e3"} Sep 16 14:32:50.590427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:50.590216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:50.623867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:50.623821 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" podStartSLOduration=2.2286828 podStartE2EDuration="2.623806113s" totalImagesPullingTime="395.123313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:50.62194992 +0000 UTC m=+1501.199311072" watchObservedRunningTime="2026-09-16 14:32:50.623806113 +0000 UTC m=+1501.201167264" Sep 16 14:32:50.721039 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:50.721002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:56.175421 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:56.175356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:56.264213 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:56.264178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:32:56.602860 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:56.602782 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:32:58.080189 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.080145 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8939-cd2655"] Sep 16 14:32:58.084031 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.084012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:32:58.086956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.086933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kdfll\"" Sep 16 14:32:58.091894 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.091870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-cd2655"] Sep 16 14:32:58.242626 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.242575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2m2\" (UniqueName: \"kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2\") pod \"test-curl-8939-cd2655\" (UID: \"bb00dea0-6721-4f08-9279-26ad0ed378ee\") " pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:32:58.343819 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.343742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2m2\" (UniqueName: \"kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2\") pod \"test-curl-8939-cd2655\" (UID: \"bb00dea0-6721-4f08-9279-26ad0ed378ee\") " pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:32:58.358994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.358967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2m2\" (UniqueName: \"kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2\") pod \"test-curl-8939-cd2655\" (UID: \"bb00dea0-6721-4f08-9279-26ad0ed378ee\") " pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:32:58.394985 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.394961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:32:58.674947 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:58.674925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-cd2655"] Sep 16 14:32:58.677770 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:32:58.677742 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb00dea0_6721_4f08_9279_26ad0ed378ee.slice/crio-ad16db2b015c145712f635c66b5d8fcbfe04f260f0c8a45b43a9f4b290d732c6 WatchSource:0}: Error finding container ad16db2b015c145712f635c66b5d8fcbfe04f260f0c8a45b43a9f4b290d732c6: Status 404 returned error can't find the container with id ad16db2b015c145712f635c66b5d8fcbfe04f260f0c8a45b43a9f4b290d732c6 Sep 16 14:32:59.648402 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:59.648368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-cd2655" event={"ID":"bb00dea0-6721-4f08-9279-26ad0ed378ee","Type":"ContainerStarted","Data":"5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad"} Sep 16 14:32:59.648402 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:59.648417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-cd2655" event={"ID":"bb00dea0-6721-4f08-9279-26ad0ed378ee","Type":"ContainerStarted","Data":"ad16db2b015c145712f635c66b5d8fcbfe04f260f0c8a45b43a9f4b290d732c6"} Sep 16 14:32:59.668226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:32:59.668183 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8939-cd2655" podStartSLOduration=1.668168614 podStartE2EDuration="1.668168614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:59.666150852 +0000 UTC m=+1510.243512004" watchObservedRunningTime="2026-09-16 14:32:59.668168614 +0000 UTC m=+1510.245529766" Sep 16 14:33:00.307135 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:00.307104 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8939-cd2655"] Sep 16 14:33:00.312377 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:00.312354 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8939-cd2655"] Sep 16 14:33:00.654875 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:00.654843 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8939-cd2655" container="test-curl-8939-cd2655" Sep 16 14:33:01.660349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:01.660308 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8939-cd2655" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" containerName="test-curl-8939-cd2655" containerID="cri-o://5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad" gracePeriod=2 Sep 16 14:33:01.662985 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:01.662957 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8939-cd2655\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:33:02.007226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:02.007187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:02.204807 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:02.204768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:02.984827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:02.984795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:03.850675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:03.850652 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:33:03.897508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:03.897480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj2m2\" (UniqueName: \"kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2\") pod \"bb00dea0-6721-4f08-9279-26ad0ed378ee\" (UID: \"bb00dea0-6721-4f08-9279-26ad0ed378ee\") " Sep 16 14:33:03.899653 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:03.899621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2" (OuterVolumeSpecName: "kube-api-access-xj2m2") pod "bb00dea0-6721-4f08-9279-26ad0ed378ee" (UID: "bb00dea0-6721-4f08-9279-26ad0ed378ee"). InnerVolumeSpecName "kube-api-access-xj2m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:33:03.998282 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:03.998245 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2m2\" (UniqueName: \"kubernetes.io/projected/bb00dea0-6721-4f08-9279-26ad0ed378ee-kube-api-access-xj2m2\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:33:04.679393 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:04.679362 2579 generic.go:356] "Generic (PLEG): container finished" podID="bb00dea0-6721-4f08-9279-26ad0ed378ee" containerID="5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad" exitCode=137 Sep 16 14:33:04.679552 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:04.679401 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:33:04.679552 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:04.679477 2579 scope.go:117] "RemoveContainer" containerID="5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad" Sep 16 14:33:04.690591 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:04.690561 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8939-cd2655\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" pod="opendatahub/test-curl-8939-cd2655" Sep 16 14:33:04.690684 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:04.690571 2579 scope.go:117] "RemoveContainer" containerID="5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad" Sep 16 14:33:04.690880 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:04.690858 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad\": container with ID starting with 5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad not found: ID does not exist" containerID="5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad" Sep 16 14:33:04.690941 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:04.690892 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad"} err="failed to get container status \"5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad\": rpc error: code = NotFound desc = could not find container \"5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad\": container with ID starting with 5c098004bb43793720d5573384733f0fbd714f536600222e7b750a39388d35ad not found: ID does not exist" Sep 16 14:33:05.818174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:05.818141 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" path="/var/lib/kubelet/pods/bb00dea0-6721-4f08-9279-26ad0ed378ee/volumes" Sep 16 14:33:09.698066 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:09.698019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:11.359927 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:11.359817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:12.164816 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:12.164782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:13.241650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:13.241595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:14.025508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:14.025461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:15.530903 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:15.530855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:15.642480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:15.642441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:16.308959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.308929 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-56ffeb"] Sep 16 14:33:16.309460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.309445 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" containerName="test-curl-8939-cd2655" Sep 16 14:33:16.309510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.309463 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" containerName="test-curl-8939-cd2655" Sep 16 14:33:16.309577 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.309528 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb00dea0-6721-4f08-9279-26ad0ed378ee" containerName="test-curl-8939-cd2655" Sep 16 14:33:16.312796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.312782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:16.316569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.316553 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kdfll\"" Sep 16 14:33:16.325796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.325773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-56ffeb"] Sep 16 14:33:16.421872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.421842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm\") pod \"test-curl-8936-56ffeb\" (UID: \"ef95b8af-7506-42fb-b3a8-f59e157f5219\") " pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:16.522835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.522804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm\") pod \"test-curl-8936-56ffeb\" (UID: \"ef95b8af-7506-42fb-b3a8-f59e157f5219\") " pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:16.531949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.531925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm\") pod \"test-curl-8936-56ffeb\" (UID: \"ef95b8af-7506-42fb-b3a8-f59e157f5219\") " pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:16.622025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.621958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:16.900726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:16.900699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-56ffeb"] Sep 16 14:33:16.902555 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:33:16.902525 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef95b8af_7506_42fb_b3a8_f59e157f5219.slice/crio-4035756b6bc8b99bc28a0918d92f3879af7d64a87aaabb11dce835c39c573668 WatchSource:0}: Error finding container 4035756b6bc8b99bc28a0918d92f3879af7d64a87aaabb11dce835c39c573668: Status 404 returned error can't find the container with id 4035756b6bc8b99bc28a0918d92f3879af7d64a87aaabb11dce835c39c573668 Sep 16 14:33:17.767491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:17.767456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-56ffeb" event={"ID":"ef95b8af-7506-42fb-b3a8-f59e157f5219","Type":"ContainerStarted","Data":"2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e"} Sep 16 14:33:17.767879 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:17.767499 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-56ffeb" event={"ID":"ef95b8af-7506-42fb-b3a8-f59e157f5219","Type":"ContainerStarted","Data":"4035756b6bc8b99bc28a0918d92f3879af7d64a87aaabb11dce835c39c573668"} Sep 16 14:33:17.788761 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:17.788714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-56ffeb" podStartSLOduration=1.7886988160000001 podStartE2EDuration="1.788698816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:17.787773339 +0000 UTC m=+1528.365134492" watchObservedRunningTime="2026-09-16 14:33:17.788698816 +0000 UTC m=+1528.366059968" Sep 16 14:33:18.405592 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.405558 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-56ffeb"] Sep 16 14:33:18.414056 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.414029 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-56ffeb"] Sep 16 14:33:18.526314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.526246 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-fe23f9"] Sep 16 14:33:18.526732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.526710 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" containerName="test-curl-8936-56ffeb" Sep 16 14:33:18.526732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.526730 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" containerName="test-curl-8936-56ffeb" Sep 16 14:33:18.526934 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.526878 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" containerName="test-curl-8936-56ffeb" Sep 16 14:33:18.530057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.530038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:18.536317 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.536282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-fe23f9"] Sep 16 14:33:18.643230 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.643200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdk24\" (UniqueName: \"kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24\") pod \"test-curl-8936-fe23f9\" (UID: \"b7cfe83a-9470-4857-991c-d7964e0b4af4\") " pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:18.744682 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.744644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdk24\" (UniqueName: \"kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24\") pod \"test-curl-8936-fe23f9\" (UID: \"b7cfe83a-9470-4857-991c-d7964e0b4af4\") " pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:18.765740 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.765710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdk24\" (UniqueName: \"kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24\") pod \"test-curl-8936-fe23f9\" (UID: \"b7cfe83a-9470-4857-991c-d7964e0b4af4\") " pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:18.774278 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:18.774257 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-56ffeb" container="test-curl-8936-56ffeb" Sep 16 14:33:18.842831 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:18.842763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:19.114422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:19.114394 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-fe23f9"] Sep 16 14:33:19.116121 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:33:19.116094 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7cfe83a_9470_4857_991c_d7964e0b4af4.slice/crio-60323a2b60a78f4b6dc180a29c4afa307c441dd3a565447bfca677937ac3cc74 WatchSource:0}: Error finding container 60323a2b60a78f4b6dc180a29c4afa307c441dd3a565447bfca677937ac3cc74: Status 404 returned error can't find the container with id 60323a2b60a78f4b6dc180a29c4afa307c441dd3a565447bfca677937ac3cc74 Sep 16 14:33:19.781502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:19.781462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-fe23f9" event={"ID":"b7cfe83a-9470-4857-991c-d7964e0b4af4","Type":"ContainerStarted","Data":"f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29"} Sep 16 14:33:19.781502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:19.781506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-fe23f9" event={"ID":"b7cfe83a-9470-4857-991c-d7964e0b4af4","Type":"ContainerStarted","Data":"60323a2b60a78f4b6dc180a29c4afa307c441dd3a565447bfca677937ac3cc74"} Sep 16 14:33:19.782035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:19.781549 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-56ffeb" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" containerName="test-curl-8936-56ffeb" containerID="cri-o://2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e" gracePeriod=2 Sep 16 14:33:19.787207 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:19.787181 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-56ffeb\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:19.810181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:19.810139 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-fe23f9" podStartSLOduration=1.81012391 podStartE2EDuration="1.81012391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:19.809584789 +0000 UTC m=+1530.386945941" watchObservedRunningTime="2026-09-16 14:33:19.81012391 +0000 UTC m=+1530.387485107" Sep 16 14:33:19.811892 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:19.811862 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-56ffeb\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:19.818943 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:19.818914 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-56ffeb\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:20.432680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.432643 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-fe23f9"] Sep 16 14:33:20.440655 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.440625 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-fe23f9"] Sep 16 14:33:20.554939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.554862 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-fb89d6"] Sep 16 14:33:20.555355 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.555341 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" containerName="test-curl-8936-fe23f9" Sep 16 14:33:20.555399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.555357 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" containerName="test-curl-8936-fe23f9" Sep 16 14:33:20.555442 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.555425 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" containerName="test-curl-8936-fe23f9" Sep 16 14:33:20.558701 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.558684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:20.566800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.566773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-fb89d6"] Sep 16 14:33:20.662003 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.661970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jsx\" (UniqueName: \"kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx\") pod \"test-curl-8936-fb89d6\" (UID: \"b4e29688-23c1-4cb4-9b24-27e9d7bd589b\") " pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:20.763278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.763243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jsx\" (UniqueName: \"kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx\") pod \"test-curl-8936-fb89d6\" (UID: \"b4e29688-23c1-4cb4-9b24-27e9d7bd589b\") " pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:20.773215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.773188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jsx\" (UniqueName: \"kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx\") pod \"test-curl-8936-fb89d6\" (UID: \"b4e29688-23c1-4cb4-9b24-27e9d7bd589b\") " pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:20.787577 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:20.787557 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-fe23f9" container="test-curl-8936-fe23f9" Sep 16 14:33:20.869855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:20.869797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:21.149214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.149179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-fb89d6"] Sep 16 14:33:21.149536 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:33:21.149510 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e29688_23c1_4cb4_9b24_27e9d7bd589b.slice/crio-a0b98f9db1b64bdf7200c4e1e967561f605d07a0bee295a8cef16c32dab63028 WatchSource:0}: Error finding container a0b98f9db1b64bdf7200c4e1e967561f605d07a0bee295a8cef16c32dab63028: Status 404 returned error can't find the container with id a0b98f9db1b64bdf7200c4e1e967561f605d07a0bee295a8cef16c32dab63028 Sep 16 14:33:21.796035 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.796002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-fb89d6" event={"ID":"b4e29688-23c1-4cb4-9b24-27e9d7bd589b","Type":"ContainerStarted","Data":"45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34"} Sep 16 14:33:21.796387 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.796053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-fb89d6" event={"ID":"b4e29688-23c1-4cb4-9b24-27e9d7bd589b","Type":"ContainerStarted","Data":"a0b98f9db1b64bdf7200c4e1e967561f605d07a0bee295a8cef16c32dab63028"} Sep 16 14:33:21.796387 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.796133 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-fe23f9" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" containerName="test-curl-8936-fe23f9" containerID="cri-o://f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29" gracePeriod=2 Sep 16 14:33:21.799521 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:21.799496 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-fe23f9\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:21.820413 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.820370 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-fb89d6" podStartSLOduration=1.820356928 podStartE2EDuration="1.820356928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:21.818449195 +0000 UTC m=+1532.395810360" watchObservedRunningTime="2026-09-16 14:33:21.820356928 +0000 UTC m=+1532.397718079" Sep 16 14:33:21.944301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:21.944274 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:22.076898 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.076821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm\") pod \"ef95b8af-7506-42fb-b3a8-f59e157f5219\" (UID: \"ef95b8af-7506-42fb-b3a8-f59e157f5219\") " Sep 16 14:33:22.078931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.078905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm" (OuterVolumeSpecName: "kube-api-access-7mftm") pod "ef95b8af-7506-42fb-b3a8-f59e157f5219" (UID: "ef95b8af-7506-42fb-b3a8-f59e157f5219"). InnerVolumeSpecName "kube-api-access-7mftm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:33:22.178456 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.178420 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/ef95b8af-7506-42fb-b3a8-f59e157f5219-kube-api-access-7mftm\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:33:22.434192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.434157 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-fb89d6"] Sep 16 14:33:22.439103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.439059 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-fb89d6"] Sep 16 14:33:22.557920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.557888 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-6e2b98"] Sep 16 14:33:22.558346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.558333 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" containerName="test-curl-8936-fb89d6" Sep 16 14:33:22.558404 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.558349 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" containerName="test-curl-8936-fb89d6" Sep 16 14:33:22.558444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.558436 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" containerName="test-curl-8936-fb89d6" Sep 16 14:33:22.561564 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.561544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:22.577197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.577166 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-6e2b98"] Sep 16 14:33:22.682783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.682740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjrl\" (UniqueName: \"kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl\") pod \"test-curl-8936-6e2b98\" (UID: \"2435be61-ff11-4dbd-8e5e-1325a66e5702\") " pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:22.784330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.784235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjrl\" (UniqueName: \"kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl\") pod \"test-curl-8936-6e2b98\" (UID: \"2435be61-ff11-4dbd-8e5e-1325a66e5702\") " pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:22.804176 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.804136 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef95b8af-7506-42fb-b3a8-f59e157f5219" containerID="2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e" exitCode=137 Sep 16 14:33:22.804644 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.804245 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:22.804644 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.804272 2579 scope.go:117] "RemoveContainer" containerID="2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e" Sep 16 14:33:22.804644 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.804361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjrl\" (UniqueName: \"kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl\") pod \"test-curl-8936-6e2b98\" (UID: \"2435be61-ff11-4dbd-8e5e-1325a66e5702\") " pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:22.804840 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:22.804687 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-fb89d6" container="test-curl-8936-fb89d6" Sep 16 14:33:22.820149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.820129 2579 scope.go:117] "RemoveContainer" containerID="2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e" Sep 16 14:33:22.820445 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:22.820425 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e\": container with ID starting with 2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e not found: ID does not exist" containerID="2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e" Sep 16 14:33:22.820521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.820454 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e"} err="failed to get container status \"2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e\": rpc error: code = NotFound desc = could not find container \"2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e\": container with ID starting with 2fd48e5757340a6ad91d6430d717bf376927ce602fa7b068223e3d8353cbeb0e not found: ID does not exist" Sep 16 14:33:22.823162 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:22.823137 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-56ffeb\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:22.876194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:22.876172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:23.161979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.161956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-6e2b98"] Sep 16 14:33:23.163793 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:33:23.163769 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2435be61_ff11_4dbd_8e5e_1325a66e5702.slice/crio-c9627c6e49a51c80864c359f584d4ec9c0260cf9e8c59b044555bb6a7cd02386 WatchSource:0}: Error finding container c9627c6e49a51c80864c359f584d4ec9c0260cf9e8c59b044555bb6a7cd02386: Status 404 returned error can't find the container with id c9627c6e49a51c80864c359f584d4ec9c0260cf9e8c59b044555bb6a7cd02386 Sep 16 14:33:23.814936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.814904 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-fb89d6" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" containerName="test-curl-8936-fb89d6" containerID="cri-o://45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34" gracePeriod=2 Sep 16 14:33:23.816246 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:23.816213 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-56ffeb\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" pod="opendatahub/test-curl-8936-56ffeb" Sep 16 14:33:23.819329 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.819308 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef95b8af-7506-42fb-b3a8-f59e157f5219" path="/var/lib/kubelet/pods/ef95b8af-7506-42fb-b3a8-f59e157f5219/volumes" Sep 16 14:33:23.819588 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.819572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-6e2b98" event={"ID":"2435be61-ff11-4dbd-8e5e-1325a66e5702","Type":"ContainerStarted","Data":"8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2"} Sep 16 14:33:23.819657 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.819625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-6e2b98" event={"ID":"2435be61-ff11-4dbd-8e5e-1325a66e5702","Type":"ContainerStarted","Data":"c9627c6e49a51c80864c359f584d4ec9c0260cf9e8c59b044555bb6a7cd02386"} Sep 16 14:33:23.840483 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.840437 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-6e2b98" podStartSLOduration=1.840422428 podStartE2EDuration="1.840422428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:23.83850772 +0000 UTC m=+1534.415868870" watchObservedRunningTime="2026-09-16 14:33:23.840422428 +0000 UTC m=+1534.417783579" Sep 16 14:33:23.982748 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:23.982722 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:24.099099 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.099016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdk24\" (UniqueName: \"kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24\") pod \"b7cfe83a-9470-4857-991c-d7964e0b4af4\" (UID: \"b7cfe83a-9470-4857-991c-d7964e0b4af4\") " Sep 16 14:33:24.101213 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.101185 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24" (OuterVolumeSpecName: "kube-api-access-pdk24") pod "b7cfe83a-9470-4857-991c-d7964e0b4af4" (UID: "b7cfe83a-9470-4857-991c-d7964e0b4af4"). InnerVolumeSpecName "kube-api-access-pdk24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:33:24.199995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.199958 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdk24\" (UniqueName: \"kubernetes.io/projected/b7cfe83a-9470-4857-991c-d7964e0b4af4-kube-api-access-pdk24\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:33:24.473042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.473011 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-6e2b98"] Sep 16 14:33:24.482700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.482673 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-6e2b98"] Sep 16 14:33:24.821731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.821651 2579 generic.go:356] "Generic (PLEG): container finished" podID="b7cfe83a-9470-4857-991c-d7964e0b4af4" containerID="f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29" exitCode=137 Sep 16 14:33:24.821731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.821690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:24.822199 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.821750 2579 scope.go:117] "RemoveContainer" containerID="f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29" Sep 16 14:33:24.822199 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:24.821951 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-6e2b98" container="test-curl-8936-6e2b98" Sep 16 14:33:24.833067 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:24.833040 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-fe23f9\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" pod="opendatahub/test-curl-8936-fe23f9" Sep 16 14:33:24.833360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.833346 2579 scope.go:117] "RemoveContainer" containerID="f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29" Sep 16 14:33:24.835755 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:24.835668 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29\": container with ID starting with f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29 not found: ID does not exist" containerID="f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29" Sep 16 14:33:24.835755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:24.835726 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29"} err="failed to get container status \"f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29\": rpc error: code = NotFound desc = could not find container \"f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29\": container with ID starting with f2c4729d2032ad88a3e834c049f12fc4704a6b5e00605468305a0ca50cc3ab29 not found: ID does not exist" Sep 16 14:33:25.117032 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:25.116938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:25.817845 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:25.817815 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7cfe83a-9470-4857-991c-d7964e0b4af4" path="/var/lib/kubelet/pods/b7cfe83a-9470-4857-991c-d7964e0b4af4/volumes" Sep 16 14:33:25.830350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:25.830318 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-6e2b98" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" containerName="test-curl-8936-6e2b98" containerID="cri-o://8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2" gracePeriod=2 Sep 16 14:33:26.006325 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.006299 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:26.117590 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.117497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6jsx\" (UniqueName: \"kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx\") pod \"b4e29688-23c1-4cb4-9b24-27e9d7bd589b\" (UID: \"b4e29688-23c1-4cb4-9b24-27e9d7bd589b\") " Sep 16 14:33:26.119745 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.119720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx" (OuterVolumeSpecName: "kube-api-access-m6jsx") pod "b4e29688-23c1-4cb4-9b24-27e9d7bd589b" (UID: "b4e29688-23c1-4cb4-9b24-27e9d7bd589b"). InnerVolumeSpecName "kube-api-access-m6jsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:33:26.218249 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.218215 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jsx\" (UniqueName: \"kubernetes.io/projected/b4e29688-23c1-4cb4-9b24-27e9d7bd589b-kube-api-access-m6jsx\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:33:26.336019 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.335985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:26.839798 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.839765 2579 generic.go:356] "Generic (PLEG): container finished" podID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" containerID="45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34" exitCode=137 Sep 16 14:33:26.840272 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.839805 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:26.840272 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.839919 2579 scope.go:117] "RemoveContainer" containerID="45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34" Sep 16 14:33:26.850568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.850549 2579 scope.go:117] "RemoveContainer" containerID="45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34" Sep 16 14:33:26.850956 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:26.850928 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34\": container with ID starting with 45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34 not found: ID does not exist" containerID="45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34" Sep 16 14:33:26.851075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:26.850963 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34"} err="failed to get container status \"45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34\": rpc error: code = NotFound desc = could not find container \"45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34\": container with ID starting with 45da6da21719f47c8fdee9ad69f232dbc0b6543fffcdd446655bad3163712c34 not found: ID does not exist" Sep 16 14:33:26.851075 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:26.851024 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-fb89d6\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" pod="opendatahub/test-curl-8936-fb89d6" Sep 16 14:33:27.819474 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:27.819429 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e29688-23c1-4cb4-9b24-27e9d7bd589b" path="/var/lib/kubelet/pods/b4e29688-23c1-4cb4-9b24-27e9d7bd589b/volumes" Sep 16 14:33:27.917388 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:27.917358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:28.023274 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.023251 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:28.139143 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.139049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkjrl\" (UniqueName: \"kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl\") pod \"2435be61-ff11-4dbd-8e5e-1325a66e5702\" (UID: \"2435be61-ff11-4dbd-8e5e-1325a66e5702\") " Sep 16 14:33:28.141234 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.141211 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl" (OuterVolumeSpecName: "kube-api-access-mkjrl") pod "2435be61-ff11-4dbd-8e5e-1325a66e5702" (UID: "2435be61-ff11-4dbd-8e5e-1325a66e5702"). InnerVolumeSpecName "kube-api-access-mkjrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:33:28.240067 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.240031 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjrl\" (UniqueName: \"kubernetes.io/projected/2435be61-ff11-4dbd-8e5e-1325a66e5702-kube-api-access-mkjrl\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:33:28.606970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.606934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:28.717578 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.717545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:28.856766 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.856729 2579 generic.go:356] "Generic (PLEG): container finished" podID="2435be61-ff11-4dbd-8e5e-1325a66e5702" containerID="8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2" exitCode=137 Sep 16 14:33:28.856945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.856790 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:28.856945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.856831 2579 scope.go:117] "RemoveContainer" containerID="8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2" Sep 16 14:33:28.867278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.867264 2579 scope.go:117] "RemoveContainer" containerID="8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2" Sep 16 14:33:28.867551 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:28.867529 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2\": container with ID starting with 8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2 not found: ID does not exist" containerID="8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2" Sep 16 14:33:28.867598 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:28.867562 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2"} err="failed to get container status \"8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2\": rpc error: code = NotFound desc = could not find container \"8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2\": container with ID starting with 8f04bea4c1199deea7014632f25b48a8e14f58a839aa67799aaee2c34dfba1d2 not found: ID does not exist" Sep 16 14:33:28.867667 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:28.867637 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-6e2b98\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:29.818536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:29.818485 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" path="/var/lib/kubelet/pods/2435be61-ff11-4dbd-8e5e-1325a66e5702/volumes" Sep 16 14:33:29.818949 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:29.818659 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-6e2b98\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" pod="opendatahub/test-curl-8936-6e2b98" Sep 16 14:33:29.992973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:29.992938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:31.421028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:31.420992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:32.411306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.411271 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:33:32.411765 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.411750 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" containerName="test-curl-8936-6e2b98" Sep 16 14:33:32.411820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.411767 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" containerName="test-curl-8936-6e2b98" Sep 16 14:33:32.411856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.411849 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2435be61-ff11-4dbd-8e5e-1325a66e5702" containerName="test-curl-8936-6e2b98" Sep 16 14:33:32.416380 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.416363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.418867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.418844 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e9829ef2-serving-cert\"" Sep 16 14:33:32.429416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.429390 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:33:32.520535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.520499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:32.580422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.580393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.580633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.580475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.580633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.580499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.580633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.580538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhzw\" (UniqueName: \"kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.619812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.619782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:32.681496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.681468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.681695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.681549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.681695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.681571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.681695 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.681637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhzw\" (UniqueName: \"kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.681695 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:32.681656 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-serving-cert: secret "maas-api-e2e-mt-e9829ef2-serving-cert" not found Sep 16 14:33:32.681936 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:33:32.681742 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls podName:a4627c46-dcdd-4645-abf7-2df0f3386cf3 nodeName:}" failed. No retries permitted until 2026-09-16 14:33:33.181722329 +0000 UTC m=+1543.759083464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls") pod "maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" (UID: "a4627c46-dcdd-4645-abf7-2df0f3386cf3") : secret "maas-api-e2e-mt-e9829ef2-serving-cert" not found Sep 16 14:33:32.682281 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.682257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.684181 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.684159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.690439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.690418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhzw\" (UniqueName: \"kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:32.896016 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:32.895981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:33.003152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.003072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:33.087423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.087387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:33.187548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.187511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:33.190386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.190360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") pod \"maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:33.326476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.326391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:33.426997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.426964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:33.610243 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.610213 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:33:33.611802 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:33:33.611774 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4627c46_dcdd_4645_abf7_2df0f3386cf3.slice/crio-187e1540524c2c9169e2e7477b90f70ad2093b1262a5b4c1f363cd18a76c1f31 WatchSource:0}: Error finding container 187e1540524c2c9169e2e7477b90f70ad2093b1262a5b4c1f363cd18a76c1f31: Status 404 returned error can't find the container with id 187e1540524c2c9169e2e7477b90f70ad2093b1262a5b4c1f363cd18a76c1f31 Sep 16 14:33:33.889519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:33.889425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" event={"ID":"a4627c46-dcdd-4645-abf7-2df0f3386cf3","Type":"ContainerStarted","Data":"187e1540524c2c9169e2e7477b90f70ad2093b1262a5b4c1f363cd18a76c1f31"} Sep 16 14:33:34.252348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:34.252310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:34.898042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:34.897933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" event={"ID":"a4627c46-dcdd-4645-abf7-2df0f3386cf3","Type":"ContainerStarted","Data":"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03"} Sep 16 14:33:34.898042 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:34.898001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:34.918075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:34.918026 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" podStartSLOduration=1.890076182 podStartE2EDuration="2.918011639s" totalImagesPullingTime="1.027935457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:34.915139871 +0000 UTC m=+1545.492501024" watchObservedRunningTime="2026-09-16 14:33:34.918011639 +0000 UTC m=+1545.495372802" Sep 16 14:33:35.220531 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:35.220500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:36.007860 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:36.007818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:40.910917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:40.910884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:33:53.550596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:53.550557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:54.811679 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:54.811649 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:33:55.024959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:55.024921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:55.767263 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:55.767212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:56.651122 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:56.651088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:57.347591 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:57.347561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:58.304591 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:58.304547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:33:58.737655 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:33:58.737593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:06.920871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:06.920828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:30.621391 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:30.621355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:35.432654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:35.432623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:35.525958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:35.525918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:37.591052 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:37.591017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:34:50.612279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:34:50.612188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:09.816158 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:09.816125 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:35:17.915255 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:17.915223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:31.165793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.165754 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8939-6d27ff"] Sep 16 14:35:31.169471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.169455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:31.172595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.172571 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kdfll\"" Sep 16 14:35:31.177883 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.177860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-6d27ff"] Sep 16 14:35:31.343547 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.343514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxkd\" (UniqueName: \"kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd\") pod \"test-curl-8939-6d27ff\" (UID: \"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d\") " pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:31.444382 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.444352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxkd\" (UniqueName: \"kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd\") pod \"test-curl-8939-6d27ff\" (UID: \"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d\") " pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:31.454021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.453988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxkd\" (UniqueName: \"kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd\") pod \"test-curl-8939-6d27ff\" (UID: \"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d\") " pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:31.479925 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.479902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:31.753842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:31.753816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-6d27ff"] Sep 16 14:35:31.755323 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:35:31.755299 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27a4df0d_2a90_4c1e_9a5e_4bac0fc4674d.slice/crio-34c86d4d0ce3a8804c2a3a1bdc7c5705f2f7664645e757ae5ad91e18d3f812e9 WatchSource:0}: Error finding container 34c86d4d0ce3a8804c2a3a1bdc7c5705f2f7664645e757ae5ad91e18d3f812e9: Status 404 returned error can't find the container with id 34c86d4d0ce3a8804c2a3a1bdc7c5705f2f7664645e757ae5ad91e18d3f812e9 Sep 16 14:35:32.646407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:32.646370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-6d27ff" event={"ID":"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d","Type":"ContainerStarted","Data":"2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003"} Sep 16 14:35:32.646407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:32.646412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-6d27ff" event={"ID":"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d","Type":"ContainerStarted","Data":"34c86d4d0ce3a8804c2a3a1bdc7c5705f2f7664645e757ae5ad91e18d3f812e9"} Sep 16 14:35:32.664778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:32.664731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8939-6d27ff" podStartSLOduration=1.664715933 podStartE2EDuration="1.664715933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:32.662210523 +0000 UTC m=+1663.239571676" watchObservedRunningTime="2026-09-16 14:35:32.664715933 +0000 UTC m=+1663.242077084" Sep 16 14:35:33.253752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.253718 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8939-6d27ff"] Sep 16 14:35:33.262306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.262280 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8939-6d27ff"] Sep 16 14:35:33.380939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.380847 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8939-1a98a4"] Sep 16 14:35:33.381422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.381404 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" containerName="test-curl-8939-6d27ff" Sep 16 14:35:33.381488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.381425 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" containerName="test-curl-8939-6d27ff" Sep 16 14:35:33.381554 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.381520 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" containerName="test-curl-8939-6d27ff" Sep 16 14:35:33.384875 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.384858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:33.390746 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.390717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-1a98a4"] Sep 16 14:35:33.564180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.564149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97l84\" (UniqueName: \"kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84\") pod \"test-curl-8939-1a98a4\" (UID: \"0030c365-52d6-45a5-84fc-83e0636f0890\") " pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:33.652645 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:33.652557 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8939-6d27ff" container="test-curl-8939-6d27ff" Sep 16 14:35:33.664761 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.664740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97l84\" (UniqueName: \"kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84\") pod \"test-curl-8939-1a98a4\" (UID: \"0030c365-52d6-45a5-84fc-83e0636f0890\") " pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:33.674081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.674057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97l84\" (UniqueName: \"kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84\") pod \"test-curl-8939-1a98a4\" (UID: \"0030c365-52d6-45a5-84fc-83e0636f0890\") " pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:33.696982 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.696964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:33.988356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:33.988332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8939-1a98a4"] Sep 16 14:35:33.990516 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:35:33.990489 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0030c365_52d6_45a5_84fc_83e0636f0890.slice/crio-a91a1ffd865ec9f415eb85b49bcc3a89e07399412a968a1ae9a94fec59df8410 WatchSource:0}: Error finding container a91a1ffd865ec9f415eb85b49bcc3a89e07399412a968a1ae9a94fec59df8410: Status 404 returned error can't find the container with id a91a1ffd865ec9f415eb85b49bcc3a89e07399412a968a1ae9a94fec59df8410 Sep 16 14:35:34.663873 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:34.663830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-1a98a4" event={"ID":"0030c365-52d6-45a5-84fc-83e0636f0890","Type":"ContainerStarted","Data":"f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129"} Sep 16 14:35:34.664338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:34.663884 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8939-1a98a4" event={"ID":"0030c365-52d6-45a5-84fc-83e0636f0890","Type":"ContainerStarted","Data":"a91a1ffd865ec9f415eb85b49bcc3a89e07399412a968a1ae9a94fec59df8410"} Sep 16 14:35:34.664338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:34.663913 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8939-6d27ff" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" containerName="test-curl-8939-6d27ff" containerID="cri-o://2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003" gracePeriod=2 Sep 16 14:35:34.681105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:34.681056 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8939-1a98a4" podStartSLOduration=1.6810392589999998 podStartE2EDuration="1.681039259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:34.679647274 +0000 UTC m=+1665.257008425" watchObservedRunningTime="2026-09-16 14:35:34.681039259 +0000 UTC m=+1665.258400455" Sep 16 14:35:35.162940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:35.162899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:35.298115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:35.298081 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8939-1a98a4"] Sep 16 14:35:35.304669 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:35.304636 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8939-1a98a4"] Sep 16 14:35:35.669983 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:35.669945 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8939-1a98a4" container="test-curl-8939-1a98a4" Sep 16 14:35:35.925902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:35.925871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:36.676782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:36.676743 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8939-1a98a4" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" containerName="test-curl-8939-1a98a4" containerID="cri-o://f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129" gracePeriod=2 Sep 16 14:35:36.853000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:36.852978 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:36.955966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:36.955934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:36.996363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:36.996266 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llxkd\" (UniqueName: \"kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd\") pod \"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d\" (UID: \"27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d\") " Sep 16 14:35:36.998670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:36.998636 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd" (OuterVolumeSpecName: "kube-api-access-llxkd") pod "27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" (UID: "27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d"). InnerVolumeSpecName "kube-api-access-llxkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:35:37.097702 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.097670 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxkd\" (UniqueName: \"kubernetes.io/projected/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d-kube-api-access-llxkd\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:37.366680 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.366583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:37.683622 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.683581 2579 generic.go:356] "Generic (PLEG): container finished" podID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" containerID="2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003" exitCode=137 Sep 16 14:35:37.684090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.683636 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:37.684090 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.683668 2579 scope.go:117] "RemoveContainer" containerID="2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003" Sep 16 14:35:37.693277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.693261 2579 scope.go:117] "RemoveContainer" containerID="2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003" Sep 16 14:35:37.693541 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:37.693524 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003\": container with ID starting with 2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003 not found: ID does not exist" containerID="2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003" Sep 16 14:35:37.693582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.693553 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003"} err="failed to get container status \"2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003\": rpc error: code = NotFound desc = could not find container \"2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003\": container with ID starting with 2ef52a3e022501bf6010a63f018d7e64500597ad3ac318dbbad62e4a2150c003 not found: ID does not exist" Sep 16 14:35:37.693993 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:37.693972 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8939-6d27ff\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" pod="opendatahub/test-curl-8939-6d27ff" Sep 16 14:35:37.814696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.814662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:37.817800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.817776 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d" path="/var/lib/kubelet/pods/27a4df0d-2a90-4c1e-9a5e-4bac0fc4674d/volumes" Sep 16 14:35:37.914349 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:37.914316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:38.457532 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:38.457498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:38.878762 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:38.878740 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:38.953321 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:38.953290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:39.019386 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.019294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97l84\" (UniqueName: \"kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84\") pod \"0030c365-52d6-45a5-84fc-83e0636f0890\" (UID: \"0030c365-52d6-45a5-84fc-83e0636f0890\") " Sep 16 14:35:39.021556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.021527 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84" (OuterVolumeSpecName: "kube-api-access-97l84") pod "0030c365-52d6-45a5-84fc-83e0636f0890" (UID: "0030c365-52d6-45a5-84fc-83e0636f0890"). InnerVolumeSpecName "kube-api-access-97l84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:35:39.120708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.120672 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97l84\" (UniqueName: \"kubernetes.io/projected/0030c365-52d6-45a5-84fc-83e0636f0890-kube-api-access-97l84\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:39.699277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.699240 2579 generic.go:356] "Generic (PLEG): container finished" podID="0030c365-52d6-45a5-84fc-83e0636f0890" containerID="f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129" exitCode=137 Sep 16 14:35:39.699479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.699290 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:39.699479 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.699343 2579 scope.go:117] "RemoveContainer" containerID="f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129" Sep 16 14:35:39.709697 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.709677 2579 scope.go:117] "RemoveContainer" containerID="f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129" Sep 16 14:35:39.709977 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:39.709960 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129\": container with ID starting with f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129 not found: ID does not exist" containerID="f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129" Sep 16 14:35:39.710043 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.709983 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129"} err="failed to get container status \"f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129\": rpc error: code = NotFound desc = could not find container \"f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129\": container with ID starting with f781fb6a14b347b8adcfb2d344427185162ab34cb346c473b682d1c4ed288129 not found: ID does not exist" Sep 16 14:35:39.710243 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:39.710222 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8939-1a98a4\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:39.820478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:39.820439 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" path="/var/lib/kubelet/pods/0030c365-52d6-45a5-84fc-83e0636f0890/volumes" Sep 16 14:35:39.827218 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:39.827178 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8939-1a98a4\" is forbidden: User \"system:node:ip-10-0-132-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-170.ec2.internal' and this object" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" pod="opendatahub/test-curl-8939-1a98a4" Sep 16 14:35:49.905764 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:49.905723 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:35:49.906210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:49.906023 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" podUID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" containerName="maas-api" containerID="cri-o://5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03" gracePeriod=30 Sep 16 14:35:49.937705 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:49.937668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:50.052555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.052516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:50.158846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.158804 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:35:50.225521 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.225482 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") pod \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " Sep 16 14:35:50.225708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.225649 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs\") pod \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " Sep 16 14:35:50.225708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.225696 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lhzw\" (UniqueName: \"kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw\") pod \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " Sep 16 14:35:50.225824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.225748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca\") pod \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\" (UID: \"a4627c46-dcdd-4645-abf7-2df0f3386cf3\") " Sep 16 14:35:50.226169 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.226142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4627c46-dcdd-4645-abf7-2df0f3386cf3" (UID: "a4627c46-dcdd-4645-abf7-2df0f3386cf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:35:50.228178 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.228152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4627c46-dcdd-4645-abf7-2df0f3386cf3" (UID: "a4627c46-dcdd-4645-abf7-2df0f3386cf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:35:50.236165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.236140 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4627c46-dcdd-4645-abf7-2df0f3386cf3" (UID: "a4627c46-dcdd-4645-abf7-2df0f3386cf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:35:50.236165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.236156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw" (OuterVolumeSpecName: "kube-api-access-4lhzw") pod "a4627c46-dcdd-4645-abf7-2df0f3386cf3" (UID: "a4627c46-dcdd-4645-abf7-2df0f3386cf3"). InnerVolumeSpecName "kube-api-access-4lhzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:35:50.271075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.271038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:50.326843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.326809 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:50.326843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.326843 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhzw\" (UniqueName: \"kubernetes.io/projected/a4627c46-dcdd-4645-abf7-2df0f3386cf3-kube-api-access-4lhzw\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:50.327185 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.326858 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4627c46-dcdd-4645-abf7-2df0f3386cf3-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:50.327185 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.326870 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4627c46-dcdd-4645-abf7-2df0f3386cf3-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:35:50.345819 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.345750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:50.771570 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.771541 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" containerID="5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03" exitCode=0 Sep 16 14:35:50.771811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.771599 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" Sep 16 14:35:50.771811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.771653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" event={"ID":"a4627c46-dcdd-4645-abf7-2df0f3386cf3","Type":"ContainerDied","Data":"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03"} Sep 16 14:35:50.771811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.771726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r" event={"ID":"a4627c46-dcdd-4645-abf7-2df0f3386cf3","Type":"ContainerDied","Data":"187e1540524c2c9169e2e7477b90f70ad2093b1262a5b4c1f363cd18a76c1f31"} Sep 16 14:35:50.771811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.771761 2579 scope.go:117] "RemoveContainer" containerID="5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03" Sep 16 14:35:50.782105 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.782089 2579 scope.go:117] "RemoveContainer" containerID="5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03" Sep 16 14:35:50.782358 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:35:50.782341 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03\": container with ID starting with 5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03 not found: ID does not exist" containerID="5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03" Sep 16 14:35:50.782405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.782367 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03"} err="failed to get container status \"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03\": rpc error: code = NotFound desc = could not find container \"5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03\": container with ID starting with 5492abc1368a8c78ba2d8cb18d9cefa61049cc7413e8d047884f0f9759dcdb03 not found: ID does not exist" Sep 16 14:35:50.808726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.808701 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:35:50.820348 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.820326 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9829ef2-84f5868869-wrk2r"] Sep 16 14:35:50.935458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:50.935423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:51.689697 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:51.689658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:51.818048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:51.818011 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" path="/var/lib/kubelet/pods/a4627c46-dcdd-4645-abf7-2df0f3386cf3/volumes" Sep 16 14:35:52.636996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:52.636946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:53.350555 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:53.350508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:53.549800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:53.549768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:57.370900 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:57.370865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:35:58.570178 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:35:58.570134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:20.810810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:20.810777 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:36:35.176045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:35.176009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:36.719902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:36.719867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:41.575460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:41.575424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:42.866953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:42.866913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:47.550467 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:47.550428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:59.222856 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:59.222811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:59.326248 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:59.326211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:36:59.534517 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:36:59.534424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:00.849950 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:00.849913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:00.954932 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:00.954901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:01.133496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:01.133387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:01.773496 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:01.773452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:01.896942 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:01.896914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:03.288253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:03.288214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:10.211601 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:10.211555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:10.298033 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:10.298001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:10.926736 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:10.926702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:17.778237 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:17.778188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:17.911956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:17.911917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:18.022804 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:18.022769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:27.811147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:27.811114 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:37:28.512524 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:28.512493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:28.737437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:28.737398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:38.652021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:38.651982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:42.530187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:42.530091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:43.518197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:43.518162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:43.965678 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:43.965641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:44.324168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:44.324080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:44.459755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:44.459722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:46.381292 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:46.381249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:47.127972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:47.127935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:47.226829 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:47.226788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:47.610627 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:47.610571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:48.490715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:48.490631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:49.334326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:49.334285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:49.751912 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:49.751884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:49.995094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:49.995062 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:37:50.007005 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.006938 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:37:50.216420 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216391 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:37:50.216901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216887 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" containerName="maas-api" Sep 16 14:37:50.216901 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216902 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" containerName="maas-api" Sep 16 14:37:50.217018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216930 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" containerName="test-curl-8939-1a98a4" Sep 16 14:37:50.217018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216936 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" containerName="test-curl-8939-1a98a4" Sep 16 14:37:50.217018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.216996 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0030c365-52d6-45a5-84fc-83e0636f0890" containerName="test-curl-8939-1a98a4" Sep 16 14:37:50.217018 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.217006 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4627c46-dcdd-4645-abf7-2df0f3386cf3" containerName="maas-api" Sep 16 14:37:50.220547 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.220527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.223544 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.223521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-363d71-serving-cert\"" Sep 16 14:37:50.234964 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.234941 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:37:50.254739 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.254698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:50.285091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.285017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qx2d\" (UniqueName: \"kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.285091 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.285058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.285271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.285124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.285271 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.285197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.382108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.382068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:50.386730 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.386702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.386848 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.386760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.386848 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.386795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qx2d\" (UniqueName: \"kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.386848 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.386819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.387034 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:37:50.386914 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-serving-cert: secret "maas-api-e2e-ipp-a-363d71-serving-cert" not found Sep 16 14:37:50.387034 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:37:50.386971 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls podName:f825c9c0-4e38-4eca-b75e-9827713bdd18 nodeName:}" failed. No retries permitted until 2026-09-16 14:37:50.886954135 +0000 UTC m=+1801.464315266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls") pod "maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18") : secret "maas-api-e2e-ipp-a-363d71-serving-cert" not found Sep 16 14:37:50.387426 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.387404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.389361 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.389342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.400012 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.399986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qx2d\" (UniqueName: \"kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.781677 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.781640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:50.887408 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.887369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:50.892403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.892374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:50.892544 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:37:50.892521 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-serving-cert: secret "maas-api-e2e-ipp-a-363d71-serving-cert" not found Sep 16 14:37:50.892588 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:37:50.892581 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls podName:f825c9c0-4e38-4eca-b75e-9827713bdd18 nodeName:}" failed. No retries permitted until 2026-09-16 14:37:51.892567065 +0000 UTC m=+1802.469928194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls") pod "maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18") : secret "maas-api-e2e-ipp-a-363d71-serving-cert" not found Sep 16 14:37:50.997315 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:50.997275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:51.332867 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:51.332821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:51.902981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:51.902942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:51.905664 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:51.905642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:52.031431 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:52.031399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:52.249427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:52.249397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:52.311302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:52.311270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:37:52.312810 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:37:52.312783 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf825c9c0_4e38_4eca_b75e_9827713bdd18.slice/crio-aad2069124835b53c600d5f99a74852943da8a4d489ace3104007d4aae51fdc9 WatchSource:0}: Error finding container aad2069124835b53c600d5f99a74852943da8a4d489ace3104007d4aae51fdc9: Status 404 returned error can't find the container with id aad2069124835b53c600d5f99a74852943da8a4d489ace3104007d4aae51fdc9 Sep 16 14:37:52.314253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:52.314233 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:37:52.533726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:52.533638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" event={"ID":"f825c9c0-4e38-4eca-b75e-9827713bdd18","Type":"ContainerStarted","Data":"aad2069124835b53c600d5f99a74852943da8a4d489ace3104007d4aae51fdc9"} Sep 16 14:37:53.541857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:53.541817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" event={"ID":"f825c9c0-4e38-4eca-b75e-9827713bdd18","Type":"ContainerStarted","Data":"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4"} Sep 16 14:37:53.542269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:53.541918 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:37:53.565516 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:53.565468 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" podStartSLOduration=3.182347188 podStartE2EDuration="3.565452618s" totalImagesPullingTime="383.10543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:37:53.562881023 +0000 UTC m=+1804.140242175" watchObservedRunningTime="2026-09-16 14:37:53.565452618 +0000 UTC m=+1804.142813812" Sep 16 14:37:56.223112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:56.223058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:56.324798 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:56.324761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:56.649872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:56.649766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:56.915450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:56.915361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:57.369164 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:57.369128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:59.450030 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:59.449995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:37:59.553747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:37:59.553720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:38:00.143439 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:00.143390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:00.276548 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:00.276508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:00.641908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:00.641877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:01.396909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:01.396872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:02.426808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:02.426771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:03.182833 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:03.182803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:03.999984 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:03.999949 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:04.005013 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.004989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.007558 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.007537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a2fada-serving-cert\"" Sep 16 14:38:04.015580 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.015558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:04.042452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.042422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:04.122752 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.122713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skn9k\" (UniqueName: \"kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.122929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.122788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.122929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.122837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.122929 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.122871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.143346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.143311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:04.224140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.224101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skn9k\" (UniqueName: \"kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.224339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.224186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.224339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.224217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.224339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.224247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.224978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.224916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.227015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.226992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.227145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.227044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.233410 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.233384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skn9k\" (UniqueName: \"kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k\") pod \"maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.244670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.244640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:04.317599 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.317527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:04.600722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.600687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:04.601175 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:38:04.601147 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247e15ea_7143_4f47_a724_4cbb67614621.slice/crio-885941f789ae1c8c78c4c9e385470d60c41d2aa4a1f1fe1d7531c9525f7824ff WatchSource:0}: Error finding container 885941f789ae1c8c78c4c9e385470d60c41d2aa4a1f1fe1d7531c9525f7824ff: Status 404 returned error can't find the container with id 885941f789ae1c8c78c4c9e385470d60c41d2aa4a1f1fe1d7531c9525f7824ff Sep 16 14:38:04.611261 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.611233 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" event={"ID":"247e15ea-7143-4f47-a724-4cbb67614621","Type":"ContainerStarted","Data":"885941f789ae1c8c78c4c9e385470d60c41d2aa4a1f1fe1d7531c9525f7824ff"} Sep 16 14:38:04.617060 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.617030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:04.715292 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:04.715261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:05.115918 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:05.115883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:05.620493 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:05.620453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" event={"ID":"247e15ea-7143-4f47-a724-4cbb67614621","Type":"ContainerStarted","Data":"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679"} Sep 16 14:38:05.620727 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:05.620526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:05.640538 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:05.640494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" podStartSLOduration=2.20881848 podStartE2EDuration="2.640477581s" totalImagesPullingTime="431.659101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:05.639253114 +0000 UTC m=+1816.216614266" watchObservedRunningTime="2026-09-16 14:38:05.640477581 +0000 UTC m=+1816.217838732" Sep 16 14:38:05.924865 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:05.924829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:06.723679 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:06.723642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:06.937949 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:06.937911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:07.616943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:07.616884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:07.723619 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:07.723568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:07.960226 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:07.960179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:08.537046 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:08.537010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:08.785808 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:08.785759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:10.829297 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:10.829261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:11.277996 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:11.277954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:11.398491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:11.398370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:11.634109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:11.634022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:11.814146 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:11.814100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:11.918520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:11.918428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:13.013524 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:13.013478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:13.779082 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:13.779048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:14.584993 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:14.584959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:14.874118 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:14.874031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:15.333341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.333288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:15.453407 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.453363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:15.737389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.737361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:38:15.741310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.741289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.744314 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.744295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-576150-openshift-default-dockercfg-xd46q\"" Sep 16 14:38:15.761881 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.761855 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:38:15.774205 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.774167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:15.842319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842560 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842560 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842560 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g86f\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842732 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842853 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.842892 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.842883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.924914 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.924872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:15.943651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.943837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.943837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.943837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.943994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.943994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.943977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944086 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g86f\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.944722 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.944703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.946383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.946363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.946651 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.946632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.953322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.953301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:15.953880 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:15.953863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g86f\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f\") pod \"e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:16.052319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.052229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:16.334731 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.334693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:38:16.335954 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:38:16.335924 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c89bc9_6cf4_41f1_8ba0_e9d292d0d011.slice/crio-2cc676656a445a36151d661b37551713dc6bd8ef81f289d51d7296104336ae90 WatchSource:0}: Error finding container 2cc676656a445a36151d661b37551713dc6bd8ef81f289d51d7296104336ae90: Status 404 returned error can't find the container with id 2cc676656a445a36151d661b37551713dc6bd8ef81f289d51d7296104336ae90 Sep 16 14:38:16.338550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.338515 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:16.338671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.338585 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:16.338671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.338638 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:16.408768 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.408739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:16.690421 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.690386 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerStarted","Data":"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061"} Sep 16 14:38:16.690597 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.690440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerStarted","Data":"2cc676656a445a36151d661b37551713dc6bd8ef81f289d51d7296104336ae90"} Sep 16 14:38:16.715797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:16.715742 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podStartSLOduration=1.715725672 podStartE2EDuration="1.715725672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:16.713847391 +0000 UTC m=+1827.291208567" watchObservedRunningTime="2026-09-16 14:38:16.715725672 +0000 UTC m=+1827.293086881" Sep 16 14:38:17.053057 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.052973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:17.058792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.058768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:17.130063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.129866 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:38:17.133973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.133945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.137758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.137739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5f0bee-serving-cert\"" Sep 16 14:38:17.147218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.147193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:38:17.211206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.211155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:17.256941 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.256911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.257095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.256958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.257095 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.257017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.257171 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.257099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lfw\" (UniqueName: \"kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.302871 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.302833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:17.358108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.358025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.358108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.358088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.358326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.358111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.358326 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.358169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lfw\" (UniqueName: \"kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.359081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.359049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.361194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.361168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.361333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.361313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.368648 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.368590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lfw\" (UniqueName: \"kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw\") pod \"maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.403083 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.403054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:17.449063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.449031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:17.577048 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.577007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:17.699101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.699068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:17.700127 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.700107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:17.737973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.737944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:38:17.738829 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:38:17.738785 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3618dec_9f77_41fe_bc30_2cbc2074c627.slice/crio-8846c38490ab270f6d5c905cd1d86f4a181f54419de5f89ec77f4bff8163b0fd WatchSource:0}: Error finding container 8846c38490ab270f6d5c905cd1d86f4a181f54419de5f89ec77f4bff8163b0fd: Status 404 returned error can't find the container with id 8846c38490ab270f6d5c905cd1d86f4a181f54419de5f89ec77f4bff8163b0fd Sep 16 14:38:17.838411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.838381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:17.938821 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:17.938778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:18.527276 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.527141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:18.708208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.708171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" event={"ID":"a3618dec-9f77-41fe-bc30-2cbc2074c627","Type":"ContainerStarted","Data":"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f"} Sep 16 14:38:18.708562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.708264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:18.708562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.708282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" event={"ID":"a3618dec-9f77-41fe-bc30-2cbc2074c627","Type":"ContainerStarted","Data":"8846c38490ab270f6d5c905cd1d86f4a181f54419de5f89ec77f4bff8163b0fd"} Sep 16 14:38:18.710140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.710114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:18.746458 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.746410 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" podStartSLOduration=1.28398869 podStartE2EDuration="1.746394665s" totalImagesPullingTime="462.405975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:18.743560354 +0000 UTC m=+1829.320921507" watchObservedRunningTime="2026-09-16 14:38:18.746394665 +0000 UTC m=+1829.323755879" Sep 16 14:38:18.808494 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:18.808418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:19.779556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:19.779519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:20.524773 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:20.524740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:21.480079 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:21.480046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:22.422139 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.421995 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:38:22.425836 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.425816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.430239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.430213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-576150-serving-cert\"" Sep 16 14:38:22.440536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.440507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:38:22.489803 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.489768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:22.508718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.508689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.508718 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.508720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2rq\" (UniqueName: \"kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.508902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.508813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.508902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.508850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.592939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.592907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:22.610063 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.610036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.610260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.610079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.610260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.610172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.610260 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:38:22.610183 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-576150-serving-cert: secret "maas-api-e2e-api-b-576150-serving-cert" not found Sep 16 14:38:22.610260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.610204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2rq\" (UniqueName: \"kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.610260 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:38:22.610239 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls podName:cbb21b34-6bc9-44ab-b138-f0b6e5a1516a nodeName:}" failed. No retries permitted until 2026-09-16 14:38:23.110223338 +0000 UTC m=+1833.687584468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls") pod "maas-api-e2e-api-b-576150-84769df7bc-qvxvq" (UID: "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a") : secret "maas-api-e2e-api-b-576150-serving-cert" not found Sep 16 14:38:22.610801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.610779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.612902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.612881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.621911 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.621892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2rq\" (UniqueName: \"kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:22.831499 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:22.831459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:23.116662 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.116553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:23.119209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.119187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") pod \"maas-api-e2e-api-b-576150-84769df7bc-qvxvq\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:23.242037 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.241999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:23.339228 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.339186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:23.361663 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.361597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:23.644842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.644815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:38:23.647006 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:38:23.646940 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb21b34_6bc9_44ab_b138_f0b6e5a1516a.slice/crio-dc5b795548e9113d735edbba304c3f8c5c0f85b1937a9bf9a471e0c18d683328 WatchSource:0}: Error finding container dc5b795548e9113d735edbba304c3f8c5c0f85b1937a9bf9a471e0c18d683328: Status 404 returned error can't find the container with id dc5b795548e9113d735edbba304c3f8c5c0f85b1937a9bf9a471e0c18d683328 Sep 16 14:38:23.751190 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.751142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" event={"ID":"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a","Type":"ContainerStarted","Data":"dc5b795548e9113d735edbba304c3f8c5c0f85b1937a9bf9a471e0c18d683328"} Sep 16 14:38:23.791977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:23.791931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:24.686812 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:24.686773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:24.721630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:24.721575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:38:24.761285 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:24.761252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" event={"ID":"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a","Type":"ContainerStarted","Data":"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7"} Sep 16 14:38:24.761493 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:24.761442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:24.782068 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:24.782020 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" podStartSLOduration=2.15151161 podStartE2EDuration="2.781999429s" totalImagesPullingTime="630.487819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:24.781357842 +0000 UTC m=+1835.358718993" watchObservedRunningTime="2026-09-16 14:38:24.781999429 +0000 UTC m=+1835.359360583" Sep 16 14:38:25.420991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:25.420954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:26.411449 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:26.411408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:27.994797 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:27.994759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:28.445846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:28.445816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:29.447848 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:29.447816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:30.247576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:30.247538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:30.773792 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:30.773756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:38:31.360921 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:31.360885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:31.594628 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:31.594588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:32.389447 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:32.389408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:33.406782 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:33.406741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:34.132844 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:34.132805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:35.846510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:35.846464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:36.854635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:36.854580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:37.727953 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:37.727919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:38.490909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:38.490868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:38.586098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:38.586064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:39.920433 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:39.920376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:40.934810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:40.934689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:41.911905 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:41.911866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:42.725218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:42.725180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:43.723512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.723477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:43.895469 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.895375 2579 generic.go:356] "Generic (PLEG): container finished" podID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerID="b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061" exitCode=0 Sep 16 14:38:43.895469 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.895455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerDied","Data":"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061"} Sep 16 14:38:43.896023 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.895962 2579 scope.go:117] "RemoveContainer" containerID="b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061" Sep 16 14:38:43.897286 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.897256 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:43.897381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.897326 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:43.897381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:43.897357 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:38:44.663534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:44.663453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:44.904208 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:44.904178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerStarted","Data":"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e"} Sep 16 14:38:45.052696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:45.052659 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:45.054331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:45.054299 2579 patch_prober.go:28] interesting pod/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:38:45.054476 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:45.054360 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Sep 16 14:38:45.726708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:45.726669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:46.053449 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:46.053354 2579 patch_prober.go:28] interesting pod/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:38:46.054098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:46.054047 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Sep 16 14:38:46.569373 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:46.569332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:47.053708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:47.053662 2579 patch_prober.go:28] interesting pod/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:38:47.054194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:47.053727 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Sep 16 14:38:47.434464 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:47.434430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:47.699459 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:47.699373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:48.053596 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:48.053498 2579 patch_prober.go:28] interesting pod/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:38:48.053806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:48.053684 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Sep 16 14:38:48.389183 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:48.389100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:49.053478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:49.053437 2579 patch_prober.go:28] interesting pod/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:38:49.053729 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:49.053513 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Sep 16 14:38:49.305661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:49.305549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:49.814517 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:49.814488 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:38:50.097927 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:50.097829 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:50.099710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:50.099688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:38:50.135015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:50.134977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:50.268582 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:50.268541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:51.628298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:51.628251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.299303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.299264 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:57.299835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.299475 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" podUID="247e15ea-7143-4f47-a724-4cbb67614621" containerName="maas-api" containerID="cri-o://f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679" gracePeriod=30 Sep 16 14:38:57.331152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.330490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.424162 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.424117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.505461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.505425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.576630 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.576578 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:57.592101 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592058 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca\") pod \"247e15ea-7143-4f47-a724-4cbb67614621\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " Sep 16 14:38:57.592269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs\") pod \"247e15ea-7143-4f47-a724-4cbb67614621\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " Sep 16 14:38:57.592269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls\") pod \"247e15ea-7143-4f47-a724-4cbb67614621\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " Sep 16 14:38:57.592269 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592245 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skn9k\" (UniqueName: \"kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k\") pod \"247e15ea-7143-4f47-a724-4cbb67614621\" (UID: \"247e15ea-7143-4f47-a724-4cbb67614621\") " Sep 16 14:38:57.592453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "247e15ea-7143-4f47-a724-4cbb67614621" (UID: "247e15ea-7143-4f47-a724-4cbb67614621"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:38:57.592688 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.592668 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e15ea-7143-4f47-a724-4cbb67614621-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:38:57.594773 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.594728 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "247e15ea-7143-4f47-a724-4cbb67614621" (UID: "247e15ea-7143-4f47-a724-4cbb67614621"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:38:57.595066 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.594849 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "247e15ea-7143-4f47-a724-4cbb67614621" (UID: "247e15ea-7143-4f47-a724-4cbb67614621"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:38:57.595066 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.594995 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k" (OuterVolumeSpecName: "kube-api-access-skn9k") pod "247e15ea-7143-4f47-a724-4cbb67614621" (UID: "247e15ea-7143-4f47-a724-4cbb67614621"). InnerVolumeSpecName "kube-api-access-skn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:38:57.604938 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.604870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.678586 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.678554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:57.693945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.693915 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:38:57.693945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.693941 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/247e15ea-7143-4f47-a724-4cbb67614621-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:38:57.693945 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.693949 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skn9k\" (UniqueName: \"kubernetes.io/projected/247e15ea-7143-4f47-a724-4cbb67614621-kube-api-access-skn9k\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:38:57.995842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.995802 2579 generic.go:356] "Generic (PLEG): container finished" podID="247e15ea-7143-4f47-a724-4cbb67614621" containerID="f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679" exitCode=0 Sep 16 14:38:57.996010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.995860 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" Sep 16 14:38:57.996010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.995884 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" event={"ID":"247e15ea-7143-4f47-a724-4cbb67614621","Type":"ContainerDied","Data":"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679"} Sep 16 14:38:57.996010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.995974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5" event={"ID":"247e15ea-7143-4f47-a724-4cbb67614621","Type":"ContainerDied","Data":"885941f789ae1c8c78c4c9e385470d60c41d2aa4a1f1fe1d7531c9525f7824ff"} Sep 16 14:38:57.996010 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:57.996002 2579 scope.go:117] "RemoveContainer" containerID="f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679" Sep 16 14:38:58.006422 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:58.006393 2579 scope.go:117] "RemoveContainer" containerID="f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679" Sep 16 14:38:58.006693 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:38:58.006668 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679\": container with ID starting with f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679 not found: ID does not exist" containerID="f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679" Sep 16 14:38:58.006769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:58.006705 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679"} err="failed to get container status \"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679\": rpc error: code = NotFound desc = could not find container \"f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679\": container with ID starting with f93dc689a239436696c9bc5e300d5de7191e40c4d56e0392d51d64112ed7f679 not found: ID does not exist" Sep 16 14:38:58.023932 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:58.023907 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:58.026747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:58.026719 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2fada-64d47dccfc-tvbn5"] Sep 16 14:38:58.152297 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:58.152201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:59.167943 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:59.167900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:38:59.818158 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:38:59.818126 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247e15ea-7143-4f47-a724-4cbb67614621" path="/var/lib/kubelet/pods/247e15ea-7143-4f47-a724-4cbb67614621/volumes" Sep 16 14:39:00.035972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:00.035936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:00.923137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:00.923093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:01.769021 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:01.768978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:02.616784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:02.616743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:03.013457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:03.013423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:03.544851 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:03.544813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:09.250140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:09.250098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:09.348001 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:09.347963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:14.526720 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:14.526596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:20.197685 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:20.197654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:20.959770 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:20.959735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:21.940403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:21.940355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:22.669715 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:22.669680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:23.653716 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:23.653674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:24.658974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:24.658941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:25.586380 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:25.586342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:26.639733 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:26.639697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:35.455030 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.454999 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:39:35.455466 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.455208 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" podUID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" containerName="maas-api" containerID="cri-o://8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7" gracePeriod=30 Sep 16 14:39:35.477159 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.477126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:35.520255 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:35.520215 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb21b34_6bc9_44ab_b138_f0b6e5a1516a.slice/crio-8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb21b34_6bc9_44ab_b138_f0b6e5a1516a.slice/crio-conmon-8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:39:35.561629 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.561577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:35.705500 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.705399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:35.723306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.723281 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:39:35.800084 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.800033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:35.873734 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.873700 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs\") pod \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " Sep 16 14:39:35.873904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.873766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp2rq\" (UniqueName: \"kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq\") pod \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " Sep 16 14:39:35.873904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.873879 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca\") pod \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " Sep 16 14:39:35.874022 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.873904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") pod \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\" (UID: \"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a\") " Sep 16 14:39:35.874241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.874207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" (UID: "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:39:35.874389 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.874326 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:35.876272 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.876239 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq" (OuterVolumeSpecName: "kube-api-access-wp2rq") pod "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" (UID: "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a"). InnerVolumeSpecName "kube-api-access-wp2rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:39:35.876681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.876653 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" (UID: "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:39:35.876778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.876679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" (UID: "cbb21b34-6bc9-44ab-b138-f0b6e5a1516a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:39:35.975510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.975417 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:35.975510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.975453 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2rq\" (UniqueName: \"kubernetes.io/projected/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-kube-api-access-wp2rq\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:35.975510 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:35.975469 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:36.072098 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.072057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:36.172450 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.172381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:36.245862 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.245773 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" containerID="8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7" exitCode=0 Sep 16 14:39:36.245862 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.245819 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" Sep 16 14:39:36.246111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.245866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" event={"ID":"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a","Type":"ContainerDied","Data":"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7"} Sep 16 14:39:36.246111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.245925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq" event={"ID":"cbb21b34-6bc9-44ab-b138-f0b6e5a1516a","Type":"ContainerDied","Data":"dc5b795548e9113d735edbba304c3f8c5c0f85b1937a9bf9a471e0c18d683328"} Sep 16 14:39:36.246111 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.245956 2579 scope.go:117] "RemoveContainer" containerID="8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7" Sep 16 14:39:36.257416 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.257399 2579 scope.go:117] "RemoveContainer" containerID="8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7" Sep 16 14:39:36.257696 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:36.257668 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7\": container with ID starting with 8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7 not found: ID does not exist" containerID="8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7" Sep 16 14:39:36.257774 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.257696 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7"} err="failed to get container status \"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7\": rpc error: code = NotFound desc = could not find container \"8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7\": container with ID starting with 8847e99958f4e41863be8771fd8089fdddb64c29db69070ff4cffa5e107bd3e7 not found: ID does not exist" Sep 16 14:39:36.269536 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.269514 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:39:36.275658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.275590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:36.277569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.277550 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-576150-84769df7bc-qvxvq"] Sep 16 14:39:36.478299 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.478258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:36.786154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:36.786113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:37.057437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:37.057353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:37.818398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:37.818367 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" path="/var/lib/kubelet/pods/cbb21b34-6bc9-44ab-b138-f0b6e5a1516a/volumes" Sep 16 14:39:37.938708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:37.938672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:38.649414 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:38.649380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:39.738843 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:39.738810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:40.314508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:40.314468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:40.416346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:40.416305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:40.841231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:40.841198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:41.701367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.701337 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:39:41.702055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702033 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" containerName="maas-api" Sep 16 14:39:41.702055 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702056 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" containerName="maas-api" Sep 16 14:39:41.702262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702069 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247e15ea-7143-4f47-a724-4cbb67614621" containerName="maas-api" Sep 16 14:39:41.702262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702074 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="247e15ea-7143-4f47-a724-4cbb67614621" containerName="maas-api" Sep 16 14:39:41.702262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702162 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb21b34-6bc9-44ab-b138-f0b6e5a1516a" containerName="maas-api" Sep 16 14:39:41.702262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.702171 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="247e15ea-7143-4f47-a724-4cbb67614621" containerName="maas-api" Sep 16 14:39:41.707145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.707121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.712768 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.712737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d5498f-serving-cert\"" Sep 16 14:39:41.725097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.725065 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:39:41.734488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.734454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.734488 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.734502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.734778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.734698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh89n\" (UniqueName: \"kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.734778 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.734727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.748819 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.748784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:41.835887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.835857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh89n\" (UniqueName: \"kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.836102 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.835895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.836180 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.836132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.836241 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.836206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.836317 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:41.836286 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-serving-cert: secret "maas-api-e2e-ipp-cleanup-d5498f-serving-cert" not found Sep 16 14:39:41.836440 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:41.836402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls podName:9545ec71-c997-476c-9784-b14218c8ad58 nodeName:}" failed. No retries permitted until 2026-09-16 14:39:42.336376014 +0000 UTC m=+1912.913737145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" (UID: "9545ec71-c997-476c-9784-b14218c8ad58") : secret "maas-api-e2e-ipp-cleanup-d5498f-serving-cert" not found Sep 16 14:39:41.836902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.836880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.839094 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.839073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.857367 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.857343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh89n\" (UniqueName: \"kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:41.862193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:41.862164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:42.090924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.090841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:42.341796 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.341715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:42.344247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.344223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:42.435158 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.435118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:42.541435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.541398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:42.618655 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.618545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:42.912034 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:42.912004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:39:42.912421 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:39:42.912397 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9545ec71_c997_476c_9784_b14218c8ad58.slice/crio-5e8321ce3b2ea4cb15430849bdc0ab8491c091c7c46bab0a5c0db1497c2ba9a5 WatchSource:0}: Error finding container 5e8321ce3b2ea4cb15430849bdc0ab8491c091c7c46bab0a5c0db1497c2ba9a5: Status 404 returned error can't find the container with id 5e8321ce3b2ea4cb15430849bdc0ab8491c091c7c46bab0a5c0db1497c2ba9a5 Sep 16 14:39:43.190902 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:43.190855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:43.292556 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:43.292523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" event={"ID":"9545ec71-c997-476c-9784-b14218c8ad58","Type":"ContainerStarted","Data":"5e8321ce3b2ea4cb15430849bdc0ab8491c091c7c46bab0a5c0db1497c2ba9a5"} Sep 16 14:39:43.976577 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:43.976540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:44.300801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.300701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" event={"ID":"9545ec71-c997-476c-9784-b14218c8ad58","Type":"ContainerStarted","Data":"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08"} Sep 16 14:39:44.300980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.300837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:44.336963 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.336913 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" podStartSLOduration=2.597733065 podStartE2EDuration="3.336898016s" totalImagesPullingTime="739.164951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:39:44.336890605 +0000 UTC m=+1914.914251758" watchObservedRunningTime="2026-09-16 14:39:44.336898016 +0000 UTC m=+1914.914259168" Sep 16 14:39:44.421277 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.421232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:44.534081 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.534047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:44.656274 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.656188 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:39:44.656429 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.656395 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" containerID="cri-o://a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e" gracePeriod=30 Sep 16 14:39:44.701218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.701189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:44.977260 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:44.977225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:46.047790 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:46.047749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:46.796064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:46.796026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:47.755041 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:47.755010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:48.683702 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:48.683658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:49.552976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:49.552937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:49.924565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:49.924541 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:39:50.021312 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021426 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021489 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021493 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.021806 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021769 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:39:50.021872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021784 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:39:50.021872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021858 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data" (OuterVolumeSpecName: "istio-data") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:39:50.021978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021955 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.022036 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.021988 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g86f\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.022435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket\") pod \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\" (UID: \"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011\") " Sep 16 14:39:50.022572 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:39:50.022572 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022393 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:39:50.022892 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022873 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istiod-ca-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.022958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022899 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.022958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022914 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-credential-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.022958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022928 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-workload-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.022958 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.022941 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-data\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.023940 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.023915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token" (OuterVolumeSpecName: "istio-token") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:39:50.024339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.024307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:39:50.024423 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.024344 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:39:50.024552 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.024532 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f" (OuterVolumeSpecName: "kube-api-access-8g86f") pod "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" (UID: "42c89bc9-6cf4-41f1-8ba0-e9d292d0d011"). InnerVolumeSpecName "kube-api-access-8g86f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:39:50.124279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.124179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.124279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.124215 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-envoy\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.124279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.124229 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-istio-podinfo\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.124279 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.124243 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g86f\" (UniqueName: \"kubernetes.io/projected/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011-kube-api-access-8g86f\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:50.313634 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.313572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:39:50.342345 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.342310 2579 generic.go:356] "Generic (PLEG): container finished" podID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerID="a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e" exitCode=0 Sep 16 14:39:50.342508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.342375 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" Sep 16 14:39:50.342508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.342396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerDied","Data":"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e"} Sep 16 14:39:50.342508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.342450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2" event={"ID":"42c89bc9-6cf4-41f1-8ba0-e9d292d0d011","Type":"ContainerDied","Data":"2cc676656a445a36151d661b37551713dc6bd8ef81f289d51d7296104336ae90"} Sep 16 14:39:50.342508 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.342473 2579 scope.go:117] "RemoveContainer" containerID="a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e" Sep 16 14:39:50.351515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.351489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:50.356477 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.356457 2579 scope.go:117] "RemoveContainer" containerID="b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061" Sep 16 14:39:50.366369 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.366350 2579 scope.go:117] "RemoveContainer" containerID="a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e" Sep 16 14:39:50.366755 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:50.366728 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e\": container with ID starting with a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e not found: ID does not exist" containerID="a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e" Sep 16 14:39:50.366876 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.366762 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e"} err="failed to get container status \"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e\": rpc error: code = NotFound desc = could not find container \"a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e\": container with ID starting with a94b2d74423ee5e2db54282a0c5bda409ccbce4504074fdbc3fb81ba8f082b3e not found: ID does not exist" Sep 16 14:39:50.366876 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.366780 2579 scope.go:117] "RemoveContainer" containerID="b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061" Sep 16 14:39:50.367096 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:50.367070 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061\": container with ID starting with b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061 not found: ID does not exist" containerID="b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061" Sep 16 14:39:50.367194 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.367109 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061"} err="failed to get container status \"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061\": rpc error: code = NotFound desc = could not find container \"b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061\": container with ID starting with b8a2cfbc86e68b3f2f6631208756b06baceddc97408894a4c5034fbef206c061 not found: ID does not exist" Sep 16 14:39:50.378579 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.378519 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:39:50.382029 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:50.382005 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-576150-openshift-default-6f78b9c7f4-wzxk2"] Sep 16 14:39:51.256224 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:51.256190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:51.818002 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:51.817972 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" path="/var/lib/kubelet/pods/42c89bc9-6cf4-41f1-8ba0-e9d292d0d011/volumes" Sep 16 14:39:55.133865 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:55.133831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:55.742923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:55.742887 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:39:55.743215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:55.743138 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" podUID="92a7eade-2b08-4759-9a3d-87f42740ace5" containerName="maas-api" containerID="cri-o://99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea" gracePeriod=30 Sep 16 14:39:55.776213 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:55.776179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:55.893959 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:55.893924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:56.003826 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.003763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:39:56.029738 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.029630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:56.085455 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.085416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvhnx\" (UniqueName: \"kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx\") pod \"92a7eade-2b08-4759-9a3d-87f42740ace5\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " Sep 16 14:39:56.085629 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.085489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs\") pod \"92a7eade-2b08-4759-9a3d-87f42740ace5\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " Sep 16 14:39:56.085629 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.085557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls\") pod \"92a7eade-2b08-4759-9a3d-87f42740ace5\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " Sep 16 14:39:56.085773 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.085632 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca\") pod \"92a7eade-2b08-4759-9a3d-87f42740ace5\" (UID: \"92a7eade-2b08-4759-9a3d-87f42740ace5\") " Sep 16 14:39:56.086032 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.085998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92a7eade-2b08-4759-9a3d-87f42740ace5" (UID: "92a7eade-2b08-4759-9a3d-87f42740ace5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:39:56.087734 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.087711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx" (OuterVolumeSpecName: "kube-api-access-wvhnx") pod "92a7eade-2b08-4759-9a3d-87f42740ace5" (UID: "92a7eade-2b08-4759-9a3d-87f42740ace5"). InnerVolumeSpecName "kube-api-access-wvhnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:39:56.088160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.088139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "92a7eade-2b08-4759-9a3d-87f42740ace5" (UID: "92a7eade-2b08-4759-9a3d-87f42740ace5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:39:56.088160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.088153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92a7eade-2b08-4759-9a3d-87f42740ace5" (UID: "92a7eade-2b08-4759-9a3d-87f42740ace5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:39:56.181565 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.181526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:56.186838 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.186816 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:56.186838 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.186838 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a7eade-2b08-4759-9a3d-87f42740ace5-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:56.186992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.186847 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhnx\" (UniqueName: \"kubernetes.io/projected/92a7eade-2b08-4759-9a3d-87f42740ace5-kube-api-access-wvhnx\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:56.186992 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.186856 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a7eade-2b08-4759-9a3d-87f42740ace5-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:39:56.382120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.382033 2579 generic.go:356] "Generic (PLEG): container finished" podID="92a7eade-2b08-4759-9a3d-87f42740ace5" containerID="99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea" exitCode=0 Sep 16 14:39:56.382120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.382077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" event={"ID":"92a7eade-2b08-4759-9a3d-87f42740ace5","Type":"ContainerDied","Data":"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea"} Sep 16 14:39:56.382120 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.382113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" event={"ID":"92a7eade-2b08-4759-9a3d-87f42740ace5","Type":"ContainerDied","Data":"b2984d34871d461a6e81ca1257a4c42269cbc9639cb8c6c0c36904dd3f8aef13"} Sep 16 14:39:56.382396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.382141 2579 scope.go:117] "RemoveContainer" containerID="99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea" Sep 16 14:39:56.382396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.382077 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94" Sep 16 14:39:56.392473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.392453 2579 scope.go:117] "RemoveContainer" containerID="99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea" Sep 16 14:39:56.392791 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:39:56.392766 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea\": container with ID starting with 99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea not found: ID does not exist" containerID="99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea" Sep 16 14:39:56.392869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.392800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea"} err="failed to get container status \"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea\": rpc error: code = NotFound desc = could not find container \"99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea\": container with ID starting with 99670709b1276806d2610166e9a100e44f204aa7f8d702cd656721d360afa8ea not found: ID does not exist" Sep 16 14:39:56.405654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.405631 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:39:56.418725 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.418707 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dc3e5b-7fc8659764-j2p94"] Sep 16 14:39:56.453220 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.453195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:56.546471 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:56.546440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:57.258766 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:57.258726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:57.811872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:57.811844 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:39:57.818134 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:57.818109 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a7eade-2b08-4759-9a3d-87f42740ace5" path="/var/lib/kubelet/pods/92a7eade-2b08-4759-9a3d-87f42740ace5/volumes" Sep 16 14:39:58.267864 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:58.267823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:39:59.221451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:39:59.221418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:00.151851 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:00.151803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:00.972738 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:00.972701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:01.861437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:01.861400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:02.746453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:02.746409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:03.732284 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:03.732216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:04.236229 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:04.236190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:04.329543 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:04.329499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:04.794936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:04.794898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:04.892888 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:04.892857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:05.840935 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:05.840895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:06.937802 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:06.937760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:07.645014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:07.644981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:08.341338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:08.341290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:09.560528 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:09.560477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:10.470264 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:10.470228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:11.535198 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:11.535160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.195405 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.195374 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:40:12.195709 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.195676 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" podUID="9545ec71-c997-476c-9784-b14218c8ad58" containerName="maas-api" containerID="cri-o://bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08" gracePeriod=30 Sep 16 14:40:12.233076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.233043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.325318 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.325287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.432250 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.432212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.459128 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.459067 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:40:12.496619 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.496566 2579 generic.go:356] "Generic (PLEG): container finished" podID="9545ec71-c997-476c-9784-b14218c8ad58" containerID="bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08" exitCode=0 Sep 16 14:40:12.496887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.496691 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" Sep 16 14:40:12.496887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.496667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" event={"ID":"9545ec71-c997-476c-9784-b14218c8ad58","Type":"ContainerDied","Data":"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08"} Sep 16 14:40:12.496887 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.496861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp" event={"ID":"9545ec71-c997-476c-9784-b14218c8ad58","Type":"ContainerDied","Data":"5e8321ce3b2ea4cb15430849bdc0ab8491c091c7c46bab0a5c0db1497c2ba9a5"} Sep 16 14:40:12.497076 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.496923 2579 scope.go:117] "RemoveContainer" containerID="bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08" Sep 16 14:40:12.507339 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.507323 2579 scope.go:117] "RemoveContainer" containerID="bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08" Sep 16 14:40:12.507698 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:40:12.507678 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08\": container with ID starting with bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08 not found: ID does not exist" containerID="bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08" Sep 16 14:40:12.507801 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.507705 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08"} err="failed to get container status \"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08\": rpc error: code = NotFound desc = could not find container \"bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08\": container with ID starting with bcafd62d13664821606f5e13ff1f6b3f7d22fcc49036bdf0c2bd11119f3b8b08 not found: ID does not exist" Sep 16 14:40:12.535147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.535114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.550461 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.550437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh89n\" (UniqueName: \"kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n\") pod \"9545ec71-c997-476c-9784-b14218c8ad58\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " Sep 16 14:40:12.551025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.550528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") pod \"9545ec71-c997-476c-9784-b14218c8ad58\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " Sep 16 14:40:12.551025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.550568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs\") pod \"9545ec71-c997-476c-9784-b14218c8ad58\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " Sep 16 14:40:12.551025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.550617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca\") pod \"9545ec71-c997-476c-9784-b14218c8ad58\" (UID: \"9545ec71-c997-476c-9784-b14218c8ad58\") " Sep 16 14:40:12.551218 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.551068 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9545ec71-c997-476c-9784-b14218c8ad58" (UID: "9545ec71-c997-476c-9784-b14218c8ad58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:40:12.553145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.553119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n" (OuterVolumeSpecName: "kube-api-access-hh89n") pod "9545ec71-c997-476c-9784-b14218c8ad58" (UID: "9545ec71-c997-476c-9784-b14218c8ad58"). InnerVolumeSpecName "kube-api-access-hh89n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:40:12.553290 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.553274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9545ec71-c997-476c-9784-b14218c8ad58" (UID: "9545ec71-c997-476c-9784-b14218c8ad58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:12.553892 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.553874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9545ec71-c997-476c-9784-b14218c8ad58" (UID: "9545ec71-c997-476c-9784-b14218c8ad58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:12.612535 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.612500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:12.651696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.651656 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:12.651696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.651682 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9545ec71-c997-476c-9784-b14218c8ad58-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:12.651696 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.651695 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9545ec71-c997-476c-9784-b14218c8ad58-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:12.651920 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.651709 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh89n\" (UniqueName: \"kubernetes.io/projected/9545ec71-c997-476c-9784-b14218c8ad58-kube-api-access-hh89n\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:12.824550 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.824523 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:40:12.837262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.837221 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5498f-7cdb84bdd8-whxjp"] Sep 16 14:40:12.916633 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:12.916573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:13.010830 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:13.010781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:13.818104 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:13.818072 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9545ec71-c997-476c-9784-b14218c8ad58" path="/var/lib/kubelet/pods/9545ec71-c997-476c-9784-b14218c8ad58/volumes" Sep 16 14:40:14.098981 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:14.098882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:14.971497 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:14.971468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:15.867463 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:15.867433 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:40:15.867663 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:15.867642 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" podUID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" containerName="maas-api" containerID="cri-o://4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37" gracePeriod=30 Sep 16 14:40:15.929595 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:15.929560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:16.042050 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.042019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:16.128756 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.128690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:40:16.149478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.149450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:16.186675 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.186598 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs\") pod \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " Sep 16 14:40:16.186855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.186775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmrg7\" (UniqueName: \"kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7\") pod \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " Sep 16 14:40:16.186930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.186860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") pod \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " Sep 16 14:40:16.186930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.186901 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca\") pod \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\" (UID: \"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e\") " Sep 16 14:40:16.187568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.187447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" (UID: "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:40:16.189500 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.189471 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" (UID: "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:16.189763 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.189729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7" (OuterVolumeSpecName: "kube-api-access-fmrg7") pod "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" (UID: "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e"). InnerVolumeSpecName "kube-api-access-fmrg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:40:16.190419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.190398 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" (UID: "21a58f2a-bd73-40ed-8d14-c03ab7a97a5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:16.287913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.287880 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrg7\" (UniqueName: \"kubernetes.io/projected/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-kube-api-access-fmrg7\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:16.287913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.287904 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:16.287913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.287914 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:16.287913 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.287923 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:16.371064 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.371034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:16.465323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.465290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:16.525599 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.525565 2579 generic.go:356] "Generic (PLEG): container finished" podID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" containerID="4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37" exitCode=0 Sep 16 14:40:16.525795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.525635 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" Sep 16 14:40:16.525795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.525657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" event={"ID":"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e","Type":"ContainerDied","Data":"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37"} Sep 16 14:40:16.525795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.525713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk" event={"ID":"21a58f2a-bd73-40ed-8d14-c03ab7a97a5e","Type":"ContainerDied","Data":"7ce4d9e44f588744ddae7b181430886dad49edd1ee32105e03303fee67fda5e3"} Sep 16 14:40:16.525795 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.525738 2579 scope.go:117] "RemoveContainer" containerID="4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37" Sep 16 14:40:16.541385 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.541365 2579 scope.go:117] "RemoveContainer" containerID="4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37" Sep 16 14:40:16.541987 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:40:16.541960 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37\": container with ID starting with 4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37 not found: ID does not exist" containerID="4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37" Sep 16 14:40:16.542152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.542122 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37"} err="failed to get container status \"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37\": rpc error: code = NotFound desc = could not find container \"4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37\": container with ID starting with 4bd8540e57b61c9f738e2a980e1311adafd7464b725a2366b804be36845fbd37 not found: ID does not exist" Sep 16 14:40:16.554930 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.554903 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:40:16.558529 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.558509 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bdfd4a-6b54f45484-pttwk"] Sep 16 14:40:16.785475 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:16.785385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:17.518186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:17.518153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:17.818534 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:17.818451 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" path="/var/lib/kubelet/pods/21a58f2a-bd73-40ed-8d14-c03ab7a97a5e/volumes" Sep 16 14:40:18.402936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:18.402887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:19.611216 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:19.611187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:20.795298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:20.795266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:21.064125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:21.064027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:21.254890 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:21.254857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:21.551517 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:21.551480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:24.493877 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:24.493830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:24.633327 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:24.633279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:24.906914 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:24.906826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:25.694666 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:25.694625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:26.629753 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:26.629717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:27.847307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:27.847269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:28.708512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:28.708480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:29.978140 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:29.978102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:40.764897 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:40.764861 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:40:40.765903 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:40.765110 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" podUID="f825c9c0-4e38-4eca-b75e-9827713bdd18" containerName="maas-api" containerID="cri-o://70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4" gracePeriod=30 Sep 16 14:40:40.811115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:40.811078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:40.971239 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:40.971202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:41.040955 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.040932 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:40:41.130835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.130801 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca\") pod \"f825c9c0-4e38-4eca-b75e-9827713bdd18\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " Sep 16 14:40:41.130994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.130860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") pod \"f825c9c0-4e38-4eca-b75e-9827713bdd18\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " Sep 16 14:40:41.130994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.130915 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qx2d\" (UniqueName: \"kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d\") pod \"f825c9c0-4e38-4eca-b75e-9827713bdd18\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " Sep 16 14:40:41.130994 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.130960 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs\") pod \"f825c9c0-4e38-4eca-b75e-9827713bdd18\" (UID: \"f825c9c0-4e38-4eca-b75e-9827713bdd18\") " Sep 16 14:40:41.131152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.131128 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f825c9c0-4e38-4eca-b75e-9827713bdd18" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:40:41.131345 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.131328 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f825c9c0-4e38-4eca-b75e-9827713bdd18-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:41.133197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.133171 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f825c9c0-4e38-4eca-b75e-9827713bdd18" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:41.133284 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.133216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f825c9c0-4e38-4eca-b75e-9827713bdd18" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:40:41.133398 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.133376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d" (OuterVolumeSpecName: "kube-api-access-7qx2d") pod "f825c9c0-4e38-4eca-b75e-9827713bdd18" (UID: "f825c9c0-4e38-4eca-b75e-9827713bdd18"). InnerVolumeSpecName "kube-api-access-7qx2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:40:41.176667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.176591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:41.232253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.232215 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:41.232253 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.232256 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qx2d\" (UniqueName: \"kubernetes.io/projected/f825c9c0-4e38-4eca-b75e-9827713bdd18-kube-api-access-7qx2d\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:41.232445 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.232271 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f825c9c0-4e38-4eca-b75e-9827713bdd18-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:40:41.305322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.305232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:41.692166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.692129 2579 generic.go:356] "Generic (PLEG): container finished" podID="f825c9c0-4e38-4eca-b75e-9827713bdd18" containerID="70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4" exitCode=0 Sep 16 14:40:41.692381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.692206 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" Sep 16 14:40:41.692381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.692229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" event={"ID":"f825c9c0-4e38-4eca-b75e-9827713bdd18","Type":"ContainerDied","Data":"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4"} Sep 16 14:40:41.692381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.692306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p" event={"ID":"f825c9c0-4e38-4eca-b75e-9827713bdd18","Type":"ContainerDied","Data":"aad2069124835b53c600d5f99a74852943da8a4d489ace3104007d4aae51fdc9"} Sep 16 14:40:41.692381 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.692328 2579 scope.go:117] "RemoveContainer" containerID="70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4" Sep 16 14:40:41.707309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.704933 2579 scope.go:117] "RemoveContainer" containerID="70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4" Sep 16 14:40:41.707309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.707010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:41.707309 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:40:41.707152 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4\": container with ID starting with 70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4 not found: ID does not exist" containerID="70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4" Sep 16 14:40:41.707309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.707190 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4"} err="failed to get container status \"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4\": rpc error: code = NotFound desc = could not find container \"70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4\": container with ID starting with 70cc24088b72f62509a7b04454614d00e403f419695aba76f2c6aff225f901d4 not found: ID does not exist" Sep 16 14:40:41.721701 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.721670 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:40:41.727452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.727423 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-363d71-7dcd77696c-r8k6p"] Sep 16 14:40:41.812765 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.812723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:41.818006 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:41.817984 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f825c9c0-4e38-4eca-b75e-9827713bdd18" path="/var/lib/kubelet/pods/f825c9c0-4e38-4eca-b75e-9827713bdd18/volumes" Sep 16 14:40:42.501846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:42.501809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:43.602615 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:43.602576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:40:46.850823 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:40:46.850787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:23.831137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:23.831103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:24.234360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:24.234305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:25.811200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:25.811169 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:41:30.161885 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:30.161849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:39.265404 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:39.265371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:44.888572 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:44.888509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:44.989872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:44.989831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:45.143977 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:45.143893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:46.370319 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:46.370284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:47.279097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:47.279058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:47.380113 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:47.380075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:47.824335 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:47.824294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:48.212286 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:48.212255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.076515 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.076472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:41:56.077096 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.076707 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" podUID="a3618dec-9f77-41fe-bc30-2cbc2074c627" containerName="maas-api" containerID="cri-o://1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f" gracePeriod=30 Sep 16 14:41:56.121069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.121034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.204811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.204740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.300893 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.300857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.334757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.334692 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:41:56.402203 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.402174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.439402 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.439374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls\") pod \"a3618dec-9f77-41fe-bc30-2cbc2074c627\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " Sep 16 14:41:56.439591 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.439463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs\") pod \"a3618dec-9f77-41fe-bc30-2cbc2074c627\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " Sep 16 14:41:56.439591 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.439529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca\") pod \"a3618dec-9f77-41fe-bc30-2cbc2074c627\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " Sep 16 14:41:56.439591 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.439571 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89lfw\" (UniqueName: \"kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw\") pod \"a3618dec-9f77-41fe-bc30-2cbc2074c627\" (UID: \"a3618dec-9f77-41fe-bc30-2cbc2074c627\") " Sep 16 14:41:56.439965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.439936 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3618dec-9f77-41fe-bc30-2cbc2074c627" (UID: "a3618dec-9f77-41fe-bc30-2cbc2074c627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:41:56.442011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.441982 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw" (OuterVolumeSpecName: "kube-api-access-89lfw") pod "a3618dec-9f77-41fe-bc30-2cbc2074c627" (UID: "a3618dec-9f77-41fe-bc30-2cbc2074c627"). InnerVolumeSpecName "kube-api-access-89lfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:41:56.442115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.442030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3618dec-9f77-41fe-bc30-2cbc2074c627" (UID: "a3618dec-9f77-41fe-bc30-2cbc2074c627"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:41:56.442115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.442054 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3618dec-9f77-41fe-bc30-2cbc2074c627" (UID: "a3618dec-9f77-41fe-bc30-2cbc2074c627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:41:56.486924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.486877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:56.540507 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.540462 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3618dec-9f77-41fe-bc30-2cbc2074c627-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:41:56.540507 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.540497 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lfw\" (UniqueName: \"kubernetes.io/projected/a3618dec-9f77-41fe-bc30-2cbc2074c627-kube-api-access-89lfw\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:41:56.540507 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.540512 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:41:56.540835 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.540527 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3618dec-9f77-41fe-bc30-2cbc2074c627-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:41:56.798889 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:56.798847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:57.033412 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.033375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:41:57.165657 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.165552 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3618dec-9f77-41fe-bc30-2cbc2074c627" containerID="1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f" exitCode=0 Sep 16 14:41:57.165657 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.165593 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" Sep 16 14:41:57.165657 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.165644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" event={"ID":"a3618dec-9f77-41fe-bc30-2cbc2074c627","Type":"ContainerDied","Data":"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f"} Sep 16 14:41:57.166175 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.165700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b" event={"ID":"a3618dec-9f77-41fe-bc30-2cbc2074c627","Type":"ContainerDied","Data":"8846c38490ab270f6d5c905cd1d86f4a181f54419de5f89ec77f4bff8163b0fd"} Sep 16 14:41:57.166175 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.165725 2579 scope.go:117] "RemoveContainer" containerID="1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f" Sep 16 14:41:57.176215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.176198 2579 scope.go:117] "RemoveContainer" containerID="1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f" Sep 16 14:41:57.176486 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:41:57.176468 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f\": container with ID starting with 1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f not found: ID does not exist" containerID="1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f" Sep 16 14:41:57.176528 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.176495 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f"} err="failed to get container status \"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f\": rpc error: code = NotFound desc = could not find container \"1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f\": container with ID starting with 1fd42648736faefbbff79b5176439da2c78a4dc584e0700642216484ab1aa87f not found: ID does not exist" Sep 16 14:41:57.201800 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.201762 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:41:57.209147 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.209124 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f0bee-bfd8cf877-pdv9b"] Sep 16 14:41:57.819221 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:57.819186 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3618dec-9f77-41fe-bc30-2cbc2074c627" path="/var/lib/kubelet/pods/a3618dec-9f77-41fe-bc30-2cbc2074c627/volumes" Sep 16 14:41:58.084401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:41:58.084312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:03.044002 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:03.043971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:03.141587 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:03.141544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:03.347267 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:03.347177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:03.960322 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:03.960274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:04.907154 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:04.907116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:05.908857 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:05.908824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:14.313519 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.313440 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:42:14.313931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.313661 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" podUID="d266b49f-3ab4-4296-9bec-cf3920316e17" containerName="maas-api" containerID="cri-o://801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e" gracePeriod=30 Sep 16 14:42:14.400937 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.400899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:14.576658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.576576 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:42:14.600793 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.600744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:14.727403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.727354 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg927\" (UniqueName: \"kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927\") pod \"d266b49f-3ab4-4296-9bec-cf3920316e17\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " Sep 16 14:42:14.727635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.727447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca\") pod \"d266b49f-3ab4-4296-9bec-cf3920316e17\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " Sep 16 14:42:14.727635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.727506 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls\") pod \"d266b49f-3ab4-4296-9bec-cf3920316e17\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " Sep 16 14:42:14.727635 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.727558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs\") pod \"d266b49f-3ab4-4296-9bec-cf3920316e17\" (UID: \"d266b49f-3ab4-4296-9bec-cf3920316e17\") " Sep 16 14:42:14.728137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.728109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d266b49f-3ab4-4296-9bec-cf3920316e17" (UID: "d266b49f-3ab4-4296-9bec-cf3920316e17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:42:14.729997 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.729972 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927" (OuterVolumeSpecName: "kube-api-access-mg927") pod "d266b49f-3ab4-4296-9bec-cf3920316e17" (UID: "d266b49f-3ab4-4296-9bec-cf3920316e17"). InnerVolumeSpecName "kube-api-access-mg927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:42:14.730356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.730324 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d266b49f-3ab4-4296-9bec-cf3920316e17" (UID: "d266b49f-3ab4-4296-9bec-cf3920316e17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:14.730356 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.730338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d266b49f-3ab4-4296-9bec-cf3920316e17" (UID: "d266b49f-3ab4-4296-9bec-cf3920316e17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:14.748909 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.748881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:14.829457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.829372 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg927\" (UniqueName: \"kubernetes.io/projected/d266b49f-3ab4-4296-9bec-cf3920316e17-kube-api-access-mg927\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:14.829457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.829410 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266b49f-3ab4-4296-9bec-cf3920316e17-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:14.829457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.829426 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:14.829457 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.829441 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266b49f-3ab4-4296-9bec-cf3920316e17-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:14.834195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:14.834133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:15.163810 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.163715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:15.261525 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.261491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:15.287931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.287894 2579 generic.go:356] "Generic (PLEG): container finished" podID="d266b49f-3ab4-4296-9bec-cf3920316e17" containerID="801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e" exitCode=0 Sep 16 14:42:15.287931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.287936 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" Sep 16 14:42:15.288152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.287974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" event={"ID":"d266b49f-3ab4-4296-9bec-cf3920316e17","Type":"ContainerDied","Data":"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e"} Sep 16 14:42:15.288152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.288026 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94" event={"ID":"d266b49f-3ab4-4296-9bec-cf3920316e17","Type":"ContainerDied","Data":"6a1cff134cbd741fe19f94a60c824fc4955592eac932b0988fd1295a792d2360"} Sep 16 14:42:15.288152 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.288047 2579 scope.go:117] "RemoveContainer" containerID="801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e" Sep 16 14:42:15.298451 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.298436 2579 scope.go:117] "RemoveContainer" containerID="801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e" Sep 16 14:42:15.298761 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:42:15.298741 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e\": container with ID starting with 801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e not found: ID does not exist" containerID="801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e" Sep 16 14:42:15.298831 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.298769 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e"} err="failed to get container status \"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e\": rpc error: code = NotFound desc = could not find container \"801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e\": container with ID starting with 801f632fdb533828f21e36ad71d8e3063f40457866c76ec7904ffc615f086a4e not found: ID does not exist" Sep 16 14:42:15.313403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.313377 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:42:15.318453 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.318426 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c0974e-5cf47c9b8c-jmh94"] Sep 16 14:42:15.817482 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:15.817444 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d266b49f-3ab4-4296-9bec-cf3920316e17" path="/var/lib/kubelet/pods/d266b49f-3ab4-4296-9bec-cf3920316e17/volumes" Sep 16 14:42:22.015437 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:22.015396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:22.120340 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:22.120306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:22.297755 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:22.297665 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:42:22.297908 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:22.297864 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" containerID="cri-o://a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af" gracePeriod=30 Sep 16 14:42:22.331309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:22.331246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:27.561112 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.561089 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:42:27.649427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649393 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649427 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649487 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649503 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649530 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chpw6\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649571 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649654 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket\") pod \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\" (UID: \"3d19d5d2-8acc-4d54-8ad7-5c72708d1690\") " Sep 16 14:42:27.649721 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649796 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649800 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.649889 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.650001 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.650054 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.650071 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istiod-ca-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.650087 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.650083 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-data\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.650641 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.650097 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-workload-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.651931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.651905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:42:27.651931 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.651922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6" (OuterVolumeSpecName: "kube-api-access-chpw6") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "kube-api-access-chpw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:42:27.652068 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.651940 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:42:27.652068 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.652014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d19d5d2-8acc-4d54-8ad7-5c72708d1690" (UID: "3d19d5d2-8acc-4d54-8ad7-5c72708d1690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:42:27.751132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.751105 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-podinfo\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.751132 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.751128 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-token\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.751278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.751138 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpw6\" (UniqueName: \"kubernetes.io/projected/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-kube-api-access-chpw6\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.751278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.751147 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-istio-envoy\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:27.751278 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:27.751155 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19d5d2-8acc-4d54-8ad7-5c72708d1690-credential-socket\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:28.376409 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.376376 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerID="a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af" exitCode=0 Sep 16 14:42:28.376585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.376421 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" Sep 16 14:42:28.376585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.376429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerDied","Data":"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af"} Sep 16 14:42:28.376585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.376470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz" event={"ID":"3d19d5d2-8acc-4d54-8ad7-5c72708d1690","Type":"ContainerDied","Data":"9580c71c05d0afac7a56df41678b21780c32df58cc2f767e0994b46aa578e4f0"} Sep 16 14:42:28.376585 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.376490 2579 scope.go:117] "RemoveContainer" containerID="a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af" Sep 16 14:42:28.388192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.388164 2579 scope.go:117] "RemoveContainer" containerID="6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12" Sep 16 14:42:28.397667 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.397639 2579 scope.go:117] "RemoveContainer" containerID="a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af" Sep 16 14:42:28.397944 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:42:28.397920 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af\": container with ID starting with a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af not found: ID does not exist" containerID="a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af" Sep 16 14:42:28.398011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.397956 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af"} err="failed to get container status \"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af\": rpc error: code = NotFound desc = could not find container \"a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af\": container with ID starting with a2514b8fe553dd0a7e7def49ca171f1253c3b0819b05bcd8e141911d9ff2f7af not found: ID does not exist" Sep 16 14:42:28.398011 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.397975 2579 scope.go:117] "RemoveContainer" containerID="6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12" Sep 16 14:42:28.398226 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:42:28.398206 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12\": container with ID starting with 6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12 not found: ID does not exist" containerID="6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12" Sep 16 14:42:28.398274 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.398235 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12"} err="failed to get container status \"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12\": rpc error: code = NotFound desc = could not find container \"6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12\": container with ID starting with 6d88cb78abddef7bc284905b23abc41b1b8e3e08df6bf98a7fa2191855f96c12 not found: ID does not exist" Sep 16 14:42:28.398432 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.398416 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:42:28.404316 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:28.404279 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c0974e-openshift-default-78d4c7ccfd-jcnmz"] Sep 16 14:42:29.818446 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:29.818416 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" path="/var/lib/kubelet/pods/3d19d5d2-8acc-4d54-8ad7-5c72708d1690/volumes" Sep 16 14:42:33.460658 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.460622 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:42:33.461040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.460829 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" podUID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" containerName="maas-api" containerID="cri-o://a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d" gracePeriod=30 Sep 16 14:42:33.475495 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.475467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:33.574976 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.574944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:33.675296 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.675257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:33.723370 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.723320 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:42:33.751074 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.751041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:33.806753 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.806724 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs\") pod \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " Sep 16 14:42:33.806923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.806772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") pod \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " Sep 16 14:42:33.806923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.806841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca\") pod \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " Sep 16 14:42:33.806923 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.806865 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg8q9\" (UniqueName: \"kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9\") pod \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\" (UID: \"0a3bfefa-6d77-43c3-8210-494a8a5271e0\") " Sep 16 14:42:33.807227 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.807203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a3bfefa-6d77-43c3-8210-494a8a5271e0" (UID: "0a3bfefa-6d77-43c3-8210-494a8a5271e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:42:33.809114 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.809084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a3bfefa-6d77-43c3-8210-494a8a5271e0" (UID: "0a3bfefa-6d77-43c3-8210-494a8a5271e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:33.809231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.809158 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a3bfefa-6d77-43c3-8210-494a8a5271e0" (UID: "0a3bfefa-6d77-43c3-8210-494a8a5271e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:33.809231 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.809154 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9" (OuterVolumeSpecName: "kube-api-access-sg8q9") pod "0a3bfefa-6d77-43c3-8210-494a8a5271e0" (UID: "0a3bfefa-6d77-43c3-8210-494a8a5271e0"). InnerVolumeSpecName "kube-api-access-sg8q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:42:33.884842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.884806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:33.908079 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.908051 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bfefa-6d77-43c3-8210-494a8a5271e0-openshift-service-ca\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:33.908079 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.908078 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8q9\" (UniqueName: \"kubernetes.io/projected/0a3bfefa-6d77-43c3-8210-494a8a5271e0-kube-api-access-sg8q9\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:33.908298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.908094 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:33.908298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:33.908109 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bfefa-6d77-43c3-8210-494a8a5271e0-maas-api-tls\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:34.125000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.124918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:34.413747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.413662 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" containerID="a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d" exitCode=0 Sep 16 14:42:34.413747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.413706 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" Sep 16 14:42:34.413979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.413748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" event={"ID":"0a3bfefa-6d77-43c3-8210-494a8a5271e0","Type":"ContainerDied","Data":"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d"} Sep 16 14:42:34.413979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.413806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w" event={"ID":"0a3bfefa-6d77-43c3-8210-494a8a5271e0","Type":"ContainerDied","Data":"80968823346d47a75bc7a22d54fa0cfc507b91e8f1632dc86a5a707098ebbf71"} Sep 16 14:42:34.413979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.413834 2579 scope.go:117] "RemoveContainer" containerID="a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d" Sep 16 14:42:34.423723 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.423703 2579 scope.go:117] "RemoveContainer" containerID="a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d" Sep 16 14:42:34.423997 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:42:34.423978 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d\": container with ID starting with a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d not found: ID does not exist" containerID="a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d" Sep 16 14:42:34.424045 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.424007 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d"} err="failed to get container status \"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d\": rpc error: code = NotFound desc = could not find container \"a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d\": container with ID starting with a11cb70b0a3daa7dfa5ff9ebc4a9f736045f136d6e5666096a03581cc61ebe5d not found: ID does not exist" Sep 16 14:42:34.433683 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.433649 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:42:34.438383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:34.438362 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d31d7c-78cf694bbb-pcb5w"] Sep 16 14:42:35.818015 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:35.817984 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" path="/var/lib/kubelet/pods/0a3bfefa-6d77-43c3-8210-494a8a5271e0/volumes" Sep 16 14:42:37.811262 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:37.811226 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:42:40.449338 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:40.449301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:40.547343 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:40.547310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:45.714568 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:45.714538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:42:50.037054 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:50.037027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:42:50.053991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:50.053966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:42:58.003333 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.003297 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:42:58.003827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.003488 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77f9796c57-k7nb2" podUID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" containerName="manager" containerID="cri-o://02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884" gracePeriod=30 Sep 16 14:42:58.258360 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.258302 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:42:58.440951 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.440919 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs\") pod \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " Sep 16 14:42:58.441138 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.441013 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert\") pod \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " Sep 16 14:42:58.441138 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.441108 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87qld\" (UniqueName: \"kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld\") pod \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\" (UID: \"5bf084cb-8674-4c38-a4ab-2b659d9df94a\") " Sep 16 14:42:58.443235 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.443204 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5bf084cb-8674-4c38-a4ab-2b659d9df94a" (UID: "5bf084cb-8674-4c38-a4ab-2b659d9df94a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:58.443354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.443251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5bf084cb-8674-4c38-a4ab-2b659d9df94a" (UID: "5bf084cb-8674-4c38-a4ab-2b659d9df94a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:42:58.443354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.443279 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld" (OuterVolumeSpecName: "kube-api-access-87qld") pod "5bf084cb-8674-4c38-a4ab-2b659d9df94a" (UID: "5bf084cb-8674-4c38-a4ab-2b659d9df94a"). InnerVolumeSpecName "kube-api-access-87qld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:42:58.542242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.542164 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-webhook-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:58.542242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.542186 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qld\" (UniqueName: \"kubernetes.io/projected/5bf084cb-8674-4c38-a4ab-2b659d9df94a-kube-api-access-87qld\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:58.542242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.542195 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf084cb-8674-4c38-a4ab-2b659d9df94a-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:42:58.559990 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.559959 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" containerID="02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884" exitCode=0 Sep 16 14:42:58.560125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.560003 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-k7nb2" Sep 16 14:42:58.560125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.560047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-k7nb2" event={"ID":"5bf084cb-8674-4c38-a4ab-2b659d9df94a","Type":"ContainerDied","Data":"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884"} Sep 16 14:42:58.560125 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.560105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-k7nb2" event={"ID":"5bf084cb-8674-4c38-a4ab-2b659d9df94a","Type":"ContainerDied","Data":"da1ef06ba188ed097c53f6d2ffdd5e4599a839ee04cacb1a3fb43660046c1480"} Sep 16 14:42:58.560255 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.560131 2579 scope.go:117] "RemoveContainer" containerID="02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884" Sep 16 14:42:58.571153 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.571130 2579 scope.go:117] "RemoveContainer" containerID="02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884" Sep 16 14:42:58.571413 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:42:58.571392 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884\": container with ID starting with 02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884 not found: ID does not exist" containerID="02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884" Sep 16 14:42:58.571462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.571421 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884"} err="failed to get container status \"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884\": rpc error: code = NotFound desc = could not find container \"02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884\": container with ID starting with 02d4d3b17d442128ae8209fe14e00f074e373a91bec32f3dcf791e5d3cc22884 not found: ID does not exist" Sep 16 14:42:58.582924 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.582896 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:42:58.587442 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:58.587423 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77f9796c57-k7nb2"] Sep 16 14:42:59.819645 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:42:59.818701 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" path="/var/lib/kubelet/pods/5bf084cb-8674-4c38-a4ab-2b659d9df94a/volumes" Sep 16 14:43:05.640670 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:05.640634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:43:51.811303 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:51.811269 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:43:59.499538 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.499497 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:43:59.500110 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500090 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" containerName="manager" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500113 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" containerName="manager" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500123 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f825c9c0-4e38-4eca-b75e-9827713bdd18" containerName="maas-api" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500132 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f825c9c0-4e38-4eca-b75e-9827713bdd18" containerName="maas-api" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500144 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500153 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500166 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500175 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:43:59.500200 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500196 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d266b49f-3ab4-4296-9bec-cf3920316e17" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500204 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d266b49f-3ab4-4296-9bec-cf3920316e17" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500214 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500222 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500230 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a7eade-2b08-4759-9a3d-87f42740ace5" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500237 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a7eade-2b08-4759-9a3d-87f42740ace5" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500246 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500253 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500261 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9545ec71-c997-476c-9784-b14218c8ad58" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500268 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9545ec71-c997-476c-9784-b14218c8ad58" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500293 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3618dec-9f77-41fe-bc30-2cbc2074c627" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500302 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3618dec-9f77-41fe-bc30-2cbc2074c627" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500399 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f825c9c0-4e38-4eca-b75e-9827713bdd18" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500410 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d266b49f-3ab4-4296-9bec-cf3920316e17" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500420 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500431 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500439 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500448 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9545ec71-c997-476c-9784-b14218c8ad58" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500458 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3bfefa-6d77-43c3-8210-494a8a5271e0" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500468 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a7eade-2b08-4759-9a3d-87f42740ace5" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500475 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3618dec-9f77-41fe-bc30-2cbc2074c627" containerName="maas-api" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500485 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf084cb-8674-4c38-a4ab-2b659d9df94a" containerName="manager" Sep 16 14:43:59.500687 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.500495 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a58f2a-bd73-40ed-8d14-c03ab7a97a5e" containerName="maas-api" Sep 16 14:43:59.503983 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.503961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.510717 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.510694 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:43:59.515368 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.515349 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:43:59.515569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.515556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nc265\"" Sep 16 14:43:59.516334 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.516316 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:43:59.531403 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.531382 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:43:59.579995 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.579968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.580109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.580003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t566\" (UniqueName: \"kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.580109 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.580063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.680947 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.680921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.681133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.680955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t566\" (UniqueName: \"kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.681133 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.680999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.683784 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.683760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.683900 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.683760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.692698 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.692678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t566\" (UniqueName: \"kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566\") pod \"maas-controller-77f9796c57-qw74f\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:43:59.817657 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:43:59.817550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:44:00.094363 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.094333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:44:00.095697 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:44:00.095664 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5804d288_2355_460d_a493_b8cb9eb386b0.slice/crio-d7fc05d070093d93739895c0ad7d7b673e247f94514e40abccc58864f5823157 WatchSource:0}: Error finding container d7fc05d070093d93739895c0ad7d7b673e247f94514e40abccc58864f5823157: Status 404 returned error can't find the container with id d7fc05d070093d93739895c0ad7d7b673e247f94514e40abccc58864f5823157 Sep 16 14:44:00.096991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.096973 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:44:00.922219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.922128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qw74f" event={"ID":"5804d288-2355-460d-a493-b8cb9eb386b0","Type":"ContainerStarted","Data":"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5"} Sep 16 14:44:00.922219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.922180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qw74f" event={"ID":"5804d288-2355-460d-a493-b8cb9eb386b0","Type":"ContainerStarted","Data":"d7fc05d070093d93739895c0ad7d7b673e247f94514e40abccc58864f5823157"} Sep 16 14:44:00.922219 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.922210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:44:00.941323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:00.941282 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77f9796c57-qw74f" podStartSLOduration=1.563350612 podStartE2EDuration="1.941268471s" totalImagesPullingTime="377.917859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:00.940160927 +0000 UTC m=+2171.517522078" watchObservedRunningTime="2026-09-16 14:44:00.941268471 +0000 UTC m=+2171.518629623" Sep 16 14:44:11.930250 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:11.930218 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:44:21.214541 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.214487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:44:21.309939 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.309894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:44:21.310462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.310424 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77f9796c57-qw74f" podUID="5804d288-2355-460d-a493-b8cb9eb386b0" containerName="manager" containerID="cri-o://41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5" gracePeriod=30 Sep 16 14:44:21.570480 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.570456 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:44:21.697354 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.697325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert\") pod \"5804d288-2355-460d-a493-b8cb9eb386b0\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " Sep 16 14:44:21.697551 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.697364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs\") pod \"5804d288-2355-460d-a493-b8cb9eb386b0\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " Sep 16 14:44:21.697551 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.697416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t566\" (UniqueName: \"kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566\") pod \"5804d288-2355-460d-a493-b8cb9eb386b0\" (UID: \"5804d288-2355-460d-a493-b8cb9eb386b0\") " Sep 16 14:44:21.699820 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.699780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5804d288-2355-460d-a493-b8cb9eb386b0" (UID: "5804d288-2355-460d-a493-b8cb9eb386b0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:44:21.700007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.699849 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5804d288-2355-460d-a493-b8cb9eb386b0" (UID: "5804d288-2355-460d-a493-b8cb9eb386b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:44:21.700007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.699864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566" (OuterVolumeSpecName: "kube-api-access-7t566") pod "5804d288-2355-460d-a493-b8cb9eb386b0" (UID: "5804d288-2355-460d-a493-b8cb9eb386b0"). InnerVolumeSpecName "kube-api-access-7t566". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:44:21.799165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.799077 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t566\" (UniqueName: \"kubernetes.io/projected/5804d288-2355-460d-a493-b8cb9eb386b0-kube-api-access-7t566\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:44:21.799165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.799107 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-webhook-cert\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:44:21.799165 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:21.799121 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5804d288-2355-460d-a493-b8cb9eb386b0-metrics-certs\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:44:22.061346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.061263 2579 generic.go:356] "Generic (PLEG): container finished" podID="5804d288-2355-460d-a493-b8cb9eb386b0" containerID="41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5" exitCode=0 Sep 16 14:44:22.061346 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.061310 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qw74f" Sep 16 14:44:22.061540 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.061353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qw74f" event={"ID":"5804d288-2355-460d-a493-b8cb9eb386b0","Type":"ContainerDied","Data":"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5"} Sep 16 14:44:22.061540 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.061410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qw74f" event={"ID":"5804d288-2355-460d-a493-b8cb9eb386b0","Type":"ContainerDied","Data":"d7fc05d070093d93739895c0ad7d7b673e247f94514e40abccc58864f5823157"} Sep 16 14:44:22.061540 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.061437 2579 scope.go:117] "RemoveContainer" containerID="41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5" Sep 16 14:44:22.070341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.070303 2579 scope.go:117] "RemoveContainer" containerID="41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5" Sep 16 14:44:22.070595 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:44:22.070579 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5\": container with ID starting with 41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5 not found: ID does not exist" containerID="41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5" Sep 16 14:44:22.070700 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.070630 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5"} err="failed to get container status \"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5\": rpc error: code = NotFound desc = could not find container \"41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5\": container with ID starting with 41793276ee322568807ea2450f1e9ce01f166e767b85152813491c8e6637d7b5 not found: ID does not exist" Sep 16 14:44:22.097294 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.097267 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:44:22.105973 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.105949 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qw74f"] Sep 16 14:44:22.926166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.926131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:44:22.938661 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.938626 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f9796c57-qdh9j"] Sep 16 14:44:22.939097 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939084 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5804d288-2355-460d-a493-b8cb9eb386b0" containerName="manager" Sep 16 14:44:22.939149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939099 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5804d288-2355-460d-a493-b8cb9eb386b0" containerName="manager" Sep 16 14:44:22.939149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939108 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:44:22.939149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939113 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d19d5d2-8acc-4d54-8ad7-5c72708d1690" containerName="istio-proxy" Sep 16 14:44:22.939149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939141 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:44:22.939149 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939147 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:44:22.939306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939224 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5804d288-2355-460d-a493-b8cb9eb386b0" containerName="manager" Sep 16 14:44:22.939306 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.939233 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c89bc9-6cf4-41f1-8ba0-e9d292d0d011" containerName="istio-proxy" Sep 16 14:44:22.943703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.943684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:22.947837 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.947811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:44:22.947980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.947813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:44:22.947980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.947831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nc265\"" Sep 16 14:44:22.947980 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.947835 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:44:22.959235 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:22.959209 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qdh9j"] Sep 16 14:44:23.110767 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.110735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-metrics-certs\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.110966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.110801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnnv\" (UniqueName: \"kubernetes.io/projected/af778b65-5f95-458c-94e2-161040708c5a-kube-api-access-blnnv\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.110966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.110829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-webhook-cert\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.211788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.211698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnnv\" (UniqueName: \"kubernetes.io/projected/af778b65-5f95-458c-94e2-161040708c5a-kube-api-access-blnnv\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.211788 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.211735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-webhook-cert\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.212025 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.211797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-metrics-certs\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.214401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.214370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-webhook-cert\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.214506 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.214414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af778b65-5f95-458c-94e2-161040708c5a-metrics-certs\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.220673 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.220648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnnv\" (UniqueName: \"kubernetes.io/projected/af778b65-5f95-458c-94e2-161040708c5a-kube-api-access-blnnv\") pod \"maas-controller-77f9796c57-qdh9j\" (UID: \"af778b65-5f95-458c-94e2-161040708c5a\") " pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.254336 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.254296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:23.531307 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.531278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f9796c57-qdh9j"] Sep 16 14:44:23.532279 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:44:23.532259 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf778b65_5f95_458c_94e2_161040708c5a.slice/crio-3a35703490dd2b63f9ce4b5df21586729dd8b42c4b2bc507a3a56d73c3f3e217 WatchSource:0}: Error finding container 3a35703490dd2b63f9ce4b5df21586729dd8b42c4b2bc507a3a56d73c3f3e217: Status 404 returned error can't find the container with id 3a35703490dd2b63f9ce4b5df21586729dd8b42c4b2bc507a3a56d73c3f3e217 Sep 16 14:44:23.817401 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:23.817319 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5804d288-2355-460d-a493-b8cb9eb386b0" path="/var/lib/kubelet/pods/5804d288-2355-460d-a493-b8cb9eb386b0/volumes" Sep 16 14:44:24.075944 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:24.075850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qdh9j" event={"ID":"af778b65-5f95-458c-94e2-161040708c5a","Type":"ContainerStarted","Data":"ff8ee595b01d8ba74d4e37cce869f03e9f2df5585ad5b016953394d958285905"} Sep 16 14:44:24.075944 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:24.075912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f9796c57-qdh9j" event={"ID":"af778b65-5f95-458c-94e2-161040708c5a","Type":"ContainerStarted","Data":"3a35703490dd2b63f9ce4b5df21586729dd8b42c4b2bc507a3a56d73c3f3e217"} Sep 16 14:44:24.076342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:24.076050 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:24.098215 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:24.098157 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77f9796c57-qdh9j" podStartSLOduration=1.74941507 podStartE2EDuration="2.098137278s" totalImagesPullingTime="348.722208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:24.097366362 +0000 UTC m=+2194.674727514" watchObservedRunningTime="2026-09-16 14:44:24.098137278 +0000 UTC m=+2194.675498431" Sep 16 14:44:35.084187 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:35.084146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f9796c57-qdh9j" Sep 16 14:44:36.628435 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:36.628407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:44:36.691917 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:36.691887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:44:57.811145 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:44:57.811110 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:45:29.626714 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:29.626677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:45:29.700000 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:29.699963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:45:37.073287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.073253 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:45:37.073991 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.073936 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" podUID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" containerName="manager" containerID="cri-o://b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad" gracePeriod=10 Sep 16 14:45:37.325970 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.325912 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:45:37.371160 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.371134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57wq4\" (UniqueName: \"kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4\") pod \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " Sep 16 14:45:37.371342 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.371237 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume\") pod \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\" (UID: \"46627e4a-0375-45c1-bdbf-1731f8ca4df2\") " Sep 16 14:45:37.371600 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.371572 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "46627e4a-0375-45c1-bdbf-1731f8ca4df2" (UID: "46627e4a-0375-45c1-bdbf-1731f8ca4df2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:45:37.373452 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.373427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4" (OuterVolumeSpecName: "kube-api-access-57wq4") pod "46627e4a-0375-45c1-bdbf-1731f8ca4df2" (UID: "46627e4a-0375-45c1-bdbf-1731f8ca4df2"). InnerVolumeSpecName "kube-api-access-57wq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:45:37.472783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.472749 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46627e4a-0375-45c1-bdbf-1731f8ca4df2-extensions-socket-volume\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:45:37.472783 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.472776 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wq4\" (UniqueName: \"kubernetes.io/projected/46627e4a-0375-45c1-bdbf-1731f8ca4df2-kube-api-access-57wq4\") on node \"ip-10-0-132-170.ec2.internal\" DevicePath \"\"" Sep 16 14:45:37.522103 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.522073 2579 generic.go:356] "Generic (PLEG): container finished" podID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" containerID="b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad" exitCode=0 Sep 16 14:45:37.522247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.522126 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" Sep 16 14:45:37.522247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.522161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" event={"ID":"46627e4a-0375-45c1-bdbf-1731f8ca4df2","Type":"ContainerDied","Data":"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad"} Sep 16 14:45:37.522247 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.522231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb" event={"ID":"46627e4a-0375-45c1-bdbf-1731f8ca4df2","Type":"ContainerDied","Data":"51b4e1e54984c0f6d19b2ece090c84384584529a4d0a6d8393cffc7387d48e81"} Sep 16 14:45:37.522399 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.522252 2579 scope.go:117] "RemoveContainer" containerID="b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad" Sep 16 14:45:37.533040 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.533022 2579 scope.go:117] "RemoveContainer" containerID="b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad" Sep 16 14:45:37.533341 ip-10-0-132-170 kubenswrapper[2579]: E0916 14:45:37.533323 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad\": container with ID starting with b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad not found: ID does not exist" containerID="b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad" Sep 16 14:45:37.533411 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.533349 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad"} err="failed to get container status \"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad\": rpc error: code = NotFound desc = could not find container \"b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad\": container with ID starting with b40f3efb646cdf1b64c509dcc191105ff2d281cf362e7d1ac64d6c437eabc6ad not found: ID does not exist" Sep 16 14:45:37.545242 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.545218 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:45:37.549460 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.549441 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8rjtb"] Sep 16 14:45:37.817455 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:37.817419 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" path="/var/lib/kubelet/pods/46627e4a-0375-45c1-bdbf-1731f8ca4df2/volumes" Sep 16 14:45:48.635956 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:48.635919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:45:48.695593 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:45:48.695546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:46:00.847383 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:00.847351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:46:22.811562 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:22.811531 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:46:37.837861 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.837831 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g"] Sep 16 14:46:37.838350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.838221 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" containerName="manager" Sep 16 14:46:37.838350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.838232 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" containerName="manager" Sep 16 14:46:37.838350 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.838311 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="46627e4a-0375-45c1-bdbf-1731f8ca4df2" containerName="manager" Sep 16 14:46:37.841462 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.841443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:37.844197 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.844175 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pnf42\"" Sep 16 14:46:37.853027 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.853003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g"] Sep 16 14:46:37.918952 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.918921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqh4\" (UniqueName: \"kubernetes.io/projected/8d970d92-e26a-47e9-bb39-263327d501d5-kube-api-access-ppqh4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:37.919137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:37.919049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d970d92-e26a-47e9-bb39-263327d501d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.019681 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.019640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqh4\" (UniqueName: \"kubernetes.io/projected/8d970d92-e26a-47e9-bb39-263327d501d5-kube-api-access-ppqh4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.019882 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.019740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d970d92-e26a-47e9-bb39-263327d501d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.020115 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.020094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d970d92-e26a-47e9-bb39-263327d501d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.033708 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.033681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqh4\" (UniqueName: \"kubernetes.io/projected/8d970d92-e26a-47e9-bb39-263327d501d5-kube-api-access-ppqh4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v2t6g\" (UID: \"8d970d92-e26a-47e9-bb39-263327d501d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.152650 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.152545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.439965 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.439940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g"] Sep 16 14:46:38.441797 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:46:38.441761 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d970d92_e26a_47e9_bb39_263327d501d5.slice/crio-a68a1ca8c7f672c65990f0d040500ff44b65818ea04c0bc2142a160073312a74 WatchSource:0}: Error finding container a68a1ca8c7f672c65990f0d040500ff44b65818ea04c0bc2142a160073312a74: Status 404 returned error can't find the container with id a68a1ca8c7f672c65990f0d040500ff44b65818ea04c0bc2142a160073312a74 Sep 16 14:46:38.873017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.872933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" event={"ID":"8d970d92-e26a-47e9-bb39-263327d501d5","Type":"ContainerStarted","Data":"f6eef87d08c9c67668b19fffac2a68605337c9e1a24cff32cf0e59a3b9ba3c28"} Sep 16 14:46:38.873017 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.872982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" event={"ID":"8d970d92-e26a-47e9-bb39-263327d501d5","Type":"ContainerStarted","Data":"a68a1ca8c7f672c65990f0d040500ff44b65818ea04c0bc2142a160073312a74"} Sep 16 14:46:38.873409 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.873018 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:46:38.895209 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:38.895160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" podStartSLOduration=1.89514569 podStartE2EDuration="1.89514569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:46:38.894713518 +0000 UTC m=+2329.472074648" watchObservedRunningTime="2026-09-16 14:46:38.89514569 +0000 UTC m=+2329.472506841" Sep 16 14:46:49.881192 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:46:49.881108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v2t6g" Sep 16 14:47:39.548759 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:47:39.548720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:47:40.811482 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:47:40.811453 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:47:50.080811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:47:50.080768 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:47:50.098916 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:47:50.098885 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:47:58.428904 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:47:58.428845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:48:45.401310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:48:45.401280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:48:46.255780 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:48:46.255742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:48:57.811137 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:48:57.811107 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:50:04.811869 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:50:04.811779 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:50:25.700936 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:50:25.700891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:50:27.172656 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:50:27.172592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:50:56.545330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:50:56.545294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:51:07.810769 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:51:07.810733 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:51:28.492353 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:51:28.492309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:52:32.665210 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.665174 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8b222/must-gather-z79zf"] Sep 16 14:52:32.669159 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.669136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.671757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.671734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8b222\"/\"kube-root-ca.crt\"" Sep 16 14:52:32.672734 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.672714 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8b222\"/\"default-dockercfg-bwccg\"" Sep 16 14:52:32.672855 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.672720 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8b222\"/\"openshift-service-ca.crt\"" Sep 16 14:52:32.687323 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.687304 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8b222/must-gather-z79zf"] Sep 16 14:52:32.700301 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.700266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:52:32.800014 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.799974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" limit=1024 Sep 16 14:52:32.819157 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.819130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/019250b5-ff23-4e3d-a738-15960bdae9e7-must-gather-output\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.819157 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.819160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8q8\" (UniqueName: \"kubernetes.io/projected/019250b5-ff23-4e3d-a738-15960bdae9e7-kube-api-access-qx8q8\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.920561 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.920473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/019250b5-ff23-4e3d-a738-15960bdae9e7-must-gather-output\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.920561 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.920506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8q8\" (UniqueName: \"kubernetes.io/projected/019250b5-ff23-4e3d-a738-15960bdae9e7-kube-api-access-qx8q8\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.920845 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.920827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/019250b5-ff23-4e3d-a738-15960bdae9e7-must-gather-output\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.928679 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.928654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8q8\" (UniqueName: \"kubernetes.io/projected/019250b5-ff23-4e3d-a738-15960bdae9e7-kube-api-access-qx8q8\") pod \"must-gather-z79zf\" (UID: \"019250b5-ff23-4e3d-a738-15960bdae9e7\") " pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:32.978697 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:32.978666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8b222/must-gather-z79zf" Sep 16 14:52:33.258329 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:33.258300 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8b222/must-gather-z79zf"] Sep 16 14:52:33.259402 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:52:33.259371 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod019250b5_ff23_4e3d_a738_15960bdae9e7.slice/crio-ad9e5ead2688da7f0765b9ad57eb114e3acbb34f0b01d006efb1a297ce3509e5 WatchSource:0}: Error finding container ad9e5ead2688da7f0765b9ad57eb114e3acbb34f0b01d006efb1a297ce3509e5: Status 404 returned error can't find the container with id ad9e5ead2688da7f0765b9ad57eb114e3acbb34f0b01d006efb1a297ce3509e5 Sep 16 14:52:33.261759 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:33.261741 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:52:33.969710 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:33.969673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8b222/must-gather-z79zf" event={"ID":"019250b5-ff23-4e3d-a738-15960bdae9e7","Type":"ContainerStarted","Data":"ad9e5ead2688da7f0765b9ad57eb114e3acbb34f0b01d006efb1a297ce3509e5"} Sep 16 14:52:35.984229 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:35.984190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8b222/must-gather-z79zf" event={"ID":"019250b5-ff23-4e3d-a738-15960bdae9e7","Type":"ContainerStarted","Data":"eebe8c16d5e3e5c6c310d263c0c99cc4e1c68ca9deb7a060bac9ef831aa46515"} Sep 16 14:52:35.984671 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:35.984248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8b222/must-gather-z79zf" event={"ID":"019250b5-ff23-4e3d-a738-15960bdae9e7","Type":"ContainerStarted","Data":"ae58d916cf2565fd944e35664298330914d34eba0dd1a3ddc7d540ca517701dc"} Sep 16 14:52:36.002123 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:36.002072 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8b222/must-gather-z79zf" podStartSLOduration=1.713955097 podStartE2EDuration="4.002054904s" totalImagesPullingTime="2.288099807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:52:35.998484825 +0000 UTC m=+2686.575845977" watchObservedRunningTime="2026-09-16 14:52:36.002054904 +0000 UTC m=+2686.579416056" Sep 16 14:52:37.130206 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:37.130173 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rbr5h_bb787601-e0a7-43ca-a6c9-6f66b529995b/global-pull-secret-syncer/0.log" Sep 16 14:52:37.187443 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:37.187414 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k76cg_e259e0c0-e3f5-4d9a-aa75-4b7716a0ed34/konnectivity-agent/0.log" Sep 16 14:52:37.286974 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:37.286941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-170.ec2.internal_579298bbec0b9e8e74fae2628d563376/haproxy/0.log" Sep 16 14:52:37.820703 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:37.820661 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-5mm9d" secret="" err="secret \"payload-processing-dockercfg-mzsgb\" not found" Sep 16 14:52:42.215169 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:42.215135 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-jdml5_55424c4e-8138-48b8-8777-e93bdc92b45f/manager/0.log" Sep 16 14:52:42.259174 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:42.259139 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8ghb5_6acdab5a-a4cd-45d3-84a0-cc23194d56eb/manager/0.log" Sep 16 14:52:42.311283 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:42.311243 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-v2t6g_8d970d92-e26a-47e9-bb39-263327d501d5/manager/0.log" Sep 16 14:52:42.414028 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:42.413996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-fzq7k_d5c49d6e-3021-4281-9eb7-e64357ab3dfa/manager/0.log" Sep 16 14:52:43.963168 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:43.963131 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-fcdzq_574a4a71-86e6-4ed6-9080-404c193e5884/cluster-monitoring-operator/0.log" Sep 16 14:52:44.076310 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.076269 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-x87gl_1d73451e-a0cb-4616-b497-844150383c23/monitoring-plugin/0.log" Sep 16 14:52:44.184549 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.184517 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4nbm_0729bf3f-a59a-474f-899d-ed06496f7884/node-exporter/0.log" Sep 16 14:52:44.211424 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.211386 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4nbm_0729bf3f-a59a-474f-899d-ed06496f7884/kube-rbac-proxy/0.log" Sep 16 14:52:44.229266 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.229187 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4nbm_0729bf3f-a59a-474f-899d-ed06496f7884/init-textfile/0.log" Sep 16 14:52:44.556351 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.556274 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wpj97_f1d2ab1c-6396-484a-a3d0-383d00ef4b5d/prometheus-operator/0.log" Sep 16 14:52:44.572491 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.572452 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wpj97_f1d2ab1c-6396-484a-a3d0-383d00ef4b5d/kube-rbac-proxy/0.log" Sep 16 14:52:44.701827 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.701800 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/thanos-query/0.log" Sep 16 14:52:44.724569 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.724542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/kube-rbac-proxy-web/0.log" Sep 16 14:52:44.747842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.747817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/kube-rbac-proxy/0.log" Sep 16 14:52:44.769166 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.769143 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/prom-label-proxy/0.log" Sep 16 14:52:44.785654 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.785626 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/kube-rbac-proxy-rules/0.log" Sep 16 14:52:44.803438 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:44.803383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769866f4f6-sh6km_117d62b8-9643-479e-ba7d-c2ee06d97f04/kube-rbac-proxy-metrics/0.log" Sep 16 14:52:46.862676 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:46.862638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:52:46.873007 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:46.872980 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/2.log" Sep 16 14:52:47.405518 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.405483 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj"] Sep 16 14:52:47.414502 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.414477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.418186 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.418156 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj"] Sep 16 14:52:47.461817 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.461771 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ff9f4c48b-n4n62_ab1abec4-ace6-4c90-b657-0d87d59aeb3c/console/0.log" Sep 16 14:52:47.477787 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.477750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-podres\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.477975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.477866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzw2s\" (UniqueName: \"kubernetes.io/projected/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-kube-api-access-pzw2s\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.477975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.477935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-sys\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.477975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.477963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-proc\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.478161 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.478000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-lib-modules\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579019 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.578977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-lib-modules\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579331 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-podres\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579444 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-podres\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579512 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-lib-modules\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579757 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzw2s\" (UniqueName: \"kubernetes.io/projected/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-kube-api-access-pzw2s\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-sys\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579846 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-proc\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-proc\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.579972 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.579939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-sys\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.588747 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.588719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzw2s\" (UniqueName: \"kubernetes.io/projected/bd027246-6f56-4e3f-ab0e-4b2b5d2513b6-kube-api-access-pzw2s\") pod \"perf-node-gather-daemonset-xs2dj\" (UID: \"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6\") " pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:47.730071 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:47.730034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:48.050745 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:48.050719 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj"] Sep 16 14:52:48.053041 ip-10-0-132-170 kubenswrapper[2579]: W0916 14:52:48.053001 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd027246_6f56_4e3f_ab0e_4b2b5d2513b6.slice/crio-fdd3a96bd64ff8cfcdb472687a375dfb9a3f304a087f575cd0f737ed853907cb WatchSource:0}: Error finding container fdd3a96bd64ff8cfcdb472687a375dfb9a3f304a087f575cd0f737ed853907cb: Status 404 returned error can't find the container with id fdd3a96bd64ff8cfcdb472687a375dfb9a3f304a087f575cd0f737ed853907cb Sep 16 14:52:48.095195 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:48.095156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" event={"ID":"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6","Type":"ContainerStarted","Data":"fdd3a96bd64ff8cfcdb472687a375dfb9a3f304a087f575cd0f737ed853907cb"} Sep 16 14:52:48.189988 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:48.189941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-sd6jz_8350b6d8-cdf2-4816-9ca1-b16f6bc0f5d1/volume-data-source-validator/0.log" Sep 16 14:52:49.105302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.105264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" event={"ID":"bd027246-6f56-4e3f-ab0e-4b2b5d2513b6","Type":"ContainerStarted","Data":"c5c90a83a60fd1e3208c47354afa3d5647445460a87f0e32718b6e1427839b6c"} Sep 16 14:52:49.105811 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.105572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:49.123966 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.123914 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" podStartSLOduration=2.123896235 podStartE2EDuration="2.123896235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:52:49.121041851 +0000 UTC m=+2699.698403016" watchObservedRunningTime="2026-09-16 14:52:49.123896235 +0000 UTC m=+2699.701257387" Sep 16 14:52:49.348862 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.348824 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmqk4_9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1/dns/0.log" Sep 16 14:52:49.365738 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.365660 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmqk4_9e6dcc4f-26d7-44fb-8585-c2e4bd2c67b1/kube-rbac-proxy/0.log" Sep 16 14:52:49.384975 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:49.384950 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4rqpt_87d538d9-4506-4269-8ec6-5d4d1c8e402a/dns-node-resolver/0.log" Sep 16 14:52:50.011576 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:50.011548 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b97t4_54f5582a-0f28-4fbd-a29b-cc8b94f968b1/node-ca/0.log" Sep 16 14:52:50.182979 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:50.182947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:52:50.191473 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:50.191439 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llzzn_d9f92411-315d-4c2e-ab0c-22332bf4c8c7/console-operator/1.log" Sep 16 14:52:51.626261 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:51.626224 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57bfccdb6c-pb265_869a4d91-529e-461c-aa3a-fec0ec4c0a0c/kube-auth-proxy/0.log" Sep 16 14:52:51.691690 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:51.691652 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-5mm9d_6b6f13b6-39b3-4135-9a21-3dee9312165f/payload-pre-processing/0.log" Sep 16 14:52:51.740520 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:51.740491 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685877fd7d-tp56g_4e8061e8-6358-4d13-bc50-c5d667cd0d4a/router/0.log" Sep 16 14:52:52.567069 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:52.567027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-66tjl_2cce6995-91cf-4a57-87ca-ee871b0751b8/serve-healthcheck-canary/0.log" Sep 16 14:52:53.065298 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:53.065267 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-z6mkc_ac63b38a-300e-483e-94b4-06db8633505f/insights-operator/0.log" Sep 16 14:52:53.074287 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:53.074253 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-z6mkc_ac63b38a-300e-483e-94b4-06db8633505f/insights-operator/1.log" Sep 16 14:52:53.149141 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:53.149111 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4sqqr_2d15e0cc-390c-4f9f-9671-02b6ca82cd9c/kube-rbac-proxy/0.log" Sep 16 14:52:53.166183 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:53.166152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4sqqr_2d15e0cc-390c-4f9f-9671-02b6ca82cd9c/exporter/0.log" Sep 16 14:52:53.182589 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:53.182525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4sqqr_2d15e0cc-390c-4f9f-9671-02b6ca82cd9c/extractor/0.log" Sep 16 14:52:55.131575 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:55.131550 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8b222/perf-node-gather-daemonset-xs2dj" Sep 16 14:52:55.304330 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:55.304294 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7z5t6_d6f0b322-e108-4804-bef5-09664f5b22eb/manager/0.log" Sep 16 14:52:55.720116 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:55.720075 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2288r_9bdbd6c4-57d5-4946-9e16-802ac7536446/manager/0.log" Sep 16 14:52:55.794362 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:55.794327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77f9796c57-qdh9j_af778b65-5f95-458c-94e2-161040708c5a/manager/0.log" Sep 16 14:52:55.811108 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:55.811050 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tdqcn_99e6de89-cdab-47c6-82ae-afece8adb6ea/server/0.log" Sep 16 14:52:56.102978 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:56.102948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d8b99777-qwkg6_c379fead-1cd2-4d3c-8327-e96957a5fb6b/manager/0.log" Sep 16 14:52:56.117214 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:56.117190 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d8b99777-qwkg6_c379fead-1cd2-4d3c-8327-e96957a5fb6b/copy-manifests/0.log" Sep 16 14:52:58.527419 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:58.527384 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57bfccdb6c-pb265_869a4d91-529e-461c-aa3a-fec0ec4c0a0c/kube-auth-proxy/0.log" Sep 16 14:52:58.573302 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:58.573276 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-5mm9d_6b6f13b6-39b3-4135-9a21-3dee9312165f/payload-pre-processing/0.log" Sep 16 14:52:58.605726 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:52:58.605692 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685877fd7d-tp56g_4e8061e8-6358-4d13-bc50-c5d667cd0d4a/router/0.log" Sep 16 14:53:00.760659 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:00.760592 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57bfccdb6c-pb265_869a4d91-529e-461c-aa3a-fec0ec4c0a0c/kube-auth-proxy/0.log" Sep 16 14:53:00.808872 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:00.808840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-5mm9d_6b6f13b6-39b3-4135-9a21-3dee9312165f/payload-pre-processing/0.log" Sep 16 14:53:00.840817 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:00.840783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685877fd7d-tp56g_4e8061e8-6358-4d13-bc50-c5d667cd0d4a/router/0.log" Sep 16 14:53:03.022577 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:03.022540 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-skt6c_efe825bb-5841-464d-9dd4-1b41c72c77a9/kube-storage-version-migrator-operator/1.log" Sep 16 14:53:03.024396 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:03.024366 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-skt6c_efe825bb-5841-464d-9dd4-1b41c72c77a9/kube-storage-version-migrator-operator/0.log" Sep 16 14:53:04.213842 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.213813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/kube-multus-additional-cni-plugins/0.log" Sep 16 14:53:04.231075 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.231039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/egress-router-binary-copy/0.log" Sep 16 14:53:04.250478 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.250451 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/cni-plugins/0.log" Sep 16 14:53:04.267824 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.267800 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/bond-cni-plugin/0.log" Sep 16 14:53:04.284309 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.284281 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/routeoverride-cni/0.log" Sep 16 14:53:04.300758 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.300707 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/whereabouts-cni-bincopy/0.log" Sep 16 14:53:04.317044 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.317017 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xh5lp_0ca46f1c-db97-4cf0-a4f1-3c4e1bafcaf5/whereabouts-cni/0.log" Sep 16 14:53:04.384193 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.384160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l8mng_e582fecb-01d3-4800-b116-f1e603c78edd/kube-multus/0.log" Sep 16 14:53:04.504019 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.503943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7lx7g_01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1/network-metrics-daemon/0.log" Sep 16 14:53:04.517341 ip-10-0-132-170 kubenswrapper[2579]: I0916 14:53:04.517318 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7lx7g_01fffcd6-2eeb-4baa-b2f5-c4d6d832a4b1/kube-rbac-proxy/0.log"